Post on 15-Dec-2015
transcript
ELIZABETH GARRETT-MAYERPROFESSOR OF BIOSTATISTICS
DIRECTOR, BIOSTATISTICS SHARED RESOURCE
HOLLINGS CANCER CENTER
Statistical Considerations for Studies Involving Mice (et al.)
What started this? ARRIVE
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research*
The gist: careful consideration should be given to planning experiments with
animals results should provide details of experimental designs, including sample
sizes, of mouse experiments.“A wealth of evidence shows that across many areas, the
reporting of biomedical research is often inadequate, leading to the view that even if the science is sound, in many cases the publications themselves are not ‘‘fit for purpose,’’ meaning that incomplete reporting of relevant information effectively renders many publications of limited value as instruments to inform policy or clinical and scientific practice.”
*Kilkenny et al., PLOS Biology, June 2010.
Why are specifics not reported?
Lots of reasons space lack of understanding of their importance lack of confidence in methods used or inability to
articulate methods recognition that results may look less important if all
details are included they are not required! peer reviewers don’t care or understand novel approaches often get panned because it isn’t a t-
test, for example.
Why are specifics not reported?
If you don’t have good rationale for your planned experimental design, what are you going to say? “we experimented on 3 mice, and the p-value was non-
significant. So, we continued adding 3 mice to the experiment until we got a significant p-value.”
“We chose 6 mice per group because we always use 6 mice per group.”
“We chose 8 mice per group because that was all the budget allowed.”
“We chose to report the results of differences in tumor size at day 45 because differences at all other days were not statistically significant.”
ARRIVE guidelines: 20 item checklist
TITLE (1) Provide as accurate and concise a description of the content of the
article as possible.ABSTRACT (2) Provide an accurate summary of the background, research objectives
(including details of the species or strain of animal used), key methods, principal findings, and conclusions of the study.
INTRODUCTION Background
(3 a). Include sufficient scientific background (including relevant references to previous work) to understand the motivation and context for the study, and explain the experimental approach and rationale.
(3 b.) Explain how and why the animal species and model being used can address the scientific objectives and, where appropriate, the study’s relevance to human biology.
Objectives (4) Clearly describe the primary and any secondary objectives of the study, or specific hypotheses being tested.
ARRIVE guidelines: 20 item checklist
METHODS Ethical statement (5) Indicate the nature of the ethical review permissions, relevant licences
(e.g. Animal [Scientific Procedures] Act 1986), and national or institutional guidelines for the care and use of animals, that cover the research.
Study design (6) For each experiment, give brief details of the study design, including: a. The number of experimental and control groups. b. Any steps taken to minimise the effects of subjective bias when allocating animals to treatment (e.g.,
randomisation procedure) and when assessing results (e.g., if done, describe who was blinded and when). c. The experimental unit (e.g. a single animal, group, or cage of animals). A time-line diagram or flow chart can
be useful to illustrate how complex study designs were carried out. Experimental procedures (7) For each experiment and each experimental group, including
controls, provide precise details of all procedures carried out. For example: a. How (e.g., drug formulation and dose, site and route of administration, anaesthesia and analgesia used
[including monitoring], surgical procedure, method of euthanasia). Provide details of any specialist equipment used, including supplier(s).
b. When (e.g., time of day). c. Where (e.g., home cage, laboratory, water maze). d. Why (e.g., rationale for choice of specific anaesthetic, route of administration, drug dose used).
Experimental animals (8) a. Provide details of the animals used, including species, strain, sex, developmental stage (e.g., mean or median
age plus age range), and weight (e.g., mean or median weight plus weight range). b. Provide further relevant information such as the source of animals, international strain nomenclature, genetic
modification status (e.g. knock-out or transgenic), genotype, health/immune status, drug- or testnaıve, previous procedures, etc.
ARRIVE guidelines: 20 item checklist
Housing and husbandry (9) Provide details of: a. Housing (e.g., type of facility, e.g., specific pathogen free (SPF); type of cage or housing;
beddingmaterial; number of cage companions; tank shape and material etc. for fish). b. Husbandry conditions (e.g., breeding programme, light/dark cycle, temperature, quality of
water etc. for fish, type of food, access to food and water, environmental enrichment). c. Welfare-related assessments and interventions that were carried out before, during, or after
the experiment. Sample size (10)
a. Specify the total number of animals used in each experiment and the number of animals in each experimental group.
b. Explain how the number of animals was decided. Provide details of any sample size calculation used.
c. Indicate the number of independent replications of each experiment, if relevant. Allocating animals to experimental groups (11)
a. Give full details of how animals were allocated to experimental groups, including randomisation or matching if done.
b. Describe the order in which the animals in the different experimental groups were treated and assessed.
Experimental outcomes (12) Clearly define the primary and secondary experimental outcomes assessed (e.g., cell death, molecular markers, behavioural changes).
ARRIVE guidelines: 20 item checklist
Statistical methods (13) a. Provide details of the statistical methods used for each analysis. b. Specify the unit of analysis for each dataset (e.g. single animal, group of animals, single
neuron). c. Describe any methods used to assess whether the data met the assumptions of the statistical
approach.
RESULTS Baseline data (14) For each experimental group, report relevant characteristics
and health status of animals (e.g., weight, microbiological status, and drug- or test-naıve) before treatment or testing (this information can often be tabulated).
Numbers analysed (15) a. Report the number of animals in each group included in each analysis. Report absolute
numbers (e.g. 10/20, not 50%a). b. If any animals or data were not included in the analysis, explain why.
Outcomes and estimation (16) Report the results for each analysis carried out, with a measure of precision (e.g., standard error or confidence interval).
Adverse events (17) a. Give details of all important adverse events in each experimental group. b. Describe any modifications to the experimental protocols made to reduce adverse events.
ARRIVE guidelines: 20 item checklist
DISCUSSION Interpretation/scientific implications (18)
a. Interpret the results, taking into account the study objectives and hypotheses, current theory, and other relevant studies in the literature.
b. Comment on the study limitations including any potential sources of bias, any limitations of the animal model, and the imprecision associated with the resultsa.
c. Describe any implications of your experimental methods or findings for the replacement, refinement, or reduction (the 3Rs) of the use of animals in research.
Generalisability/translation (19) Comment on whether, and how, the findings of this study are likely to
translate to other species or systems, including any relevance to human biology.
Funding (20) List all funding sources (including grant number) and the role of the
funder(s) in the study.
Manuscripts vs. grants
Similar principles, but very limited space in grants
Convince the reviewers you have a clear question you have a method for gathering relevant data you can measure the outcomes of interest you know what to do with the data the results will answer the clear question
1. You have a clear question
Stated objective and rationale should make it clear what your scientific question is.
Stating your hypothesis can be very helpful. E.g., the knockout model will have faster tumor
growth than the wildtype model. E.g., gene expression will be lower in the mice treated
with inhibitor compared to untreated mice.This is standard in clinical research but not
as common in basic/translational research.
2. You have a method for gathering relevant data
Experimental design!!Details….we need details.If you cannot explain the design to the reviewer, he
cannot understand the data that is generated.Be very careful of ‘bias’s’ in your designExample: evaluating metastases.
design says mice will be followed until death or primary tumor reaches xx mm3.
Differential follow-up time. Example: comparing tumor size
design says you will compare tumor volumes at day 60 preliminary data suggests that you will have had to sacrifice most of
the animals in your control group by day 50. how can you compare tumors when the mouse died 10+ days ago?
2. You have a method for gathering relevant data
Randomization? important to consider “confounders” confounder: a ‘variable’ that might affect your outcome
that is not related to the experimental conditions e.g., shipping batch e.g., diet/temperature/location of cage
Blinding? inherent biases when you know group assignment! important to NOT know if the mouse being evaluated is
in the group you expect to do better/worse. subconcious effects. in clinical research, taken for
granted.
2. You have a method for gathering relevant data
Sources of variation? transgenic vs. xenografts models? how many cell lines? Or, are you using (multiple)
primary tumors? fresh vs. frozen tissue? reference gene? is it really ‘stable’? Sampling
longitudinal? same mouse measured repeatedly over time separate cohorts? sac’ing different cohorts over time
Measurement same vs. different ‘raters’? different measurement approaches?
3. how you measure the outcome
Very often see “we will evaluate differences in antitumor efficacy”
tumor size at time t? tumor growth rate? presence/absence of metastases? tumor take rate? caliper vs. imaging measures
Other measures: gene expression, methylation, mutations, histology, etc. assays need to be included type of measure should be included
continuous expression? positive vs. negative? IHC, for example, can be expressed in many ways.
4. You know what to do with data
Example: You have measured the tumor volume every 5 days for 100 days
on 100 mice. 21 measures per mouse x 100 mice = 2100 measures. Why would you look only at the measures at one time point?
Statistical analysis plan is very important.There are different approaches for
continuous outcomes (e.g. tumor volume at time t). binary/categorical outcomes (e.g, presence of metastases) time to event outcomes (e.g., time to death/sacrifice)
There are different approaches for longitudinal data comparing vs. estimating vs. dose finding
Example
Moussa O, Ashton AW, Fraig M, Garrett-Mayer E, Ghoneim MA, Halushka PV, Watson DK. Novel role of thromboxane receptors beta isoform in bladder cancer pathogenesis. Cancer Research, 2008, Jun 1; 68(11): 4097-4104.
Xenograft mouse model. TCC-SUP tumorigenic human bladder cancer cells were selected as they express TP-β receptor and were used for the drug combination studies. Immortalized nontransformed normal urothelial SV-HUC cells were selected because they express the TP-α. These cells were stably transfected with pcDNA3, TP-α, or TP-β for cell transformation studies. Both cell lines were used in a s.c. model in immunocompromised (nu/nu) mice. TCC-SUP cells (5 × 106) or SV-HUC cells (5 × 107) in Matrigel (BD Bioscience, Inc.) were injected s.c. into the right and left flanks of anesthetized mice. Tumor growth was monitored in these mice twice a week. For mice injected with TCC-SUP, GR32191 or vehicle control was administered daily (20 mg/kg) by gavage with treatment initiated 24 h after initial injection. Two cycles of cisplatin [single high dose (5 mg/kg) or single low dose (0.5 mg/kg)] were administrated at day 4 and day 11 post-tumor cell injection
The data
0 20 40 60 80
Time (days)
Vo
lum
e
0
30100
300
600
1000
2000
3000
4000 High Cis+GRHigh CisLow Cis+GRLow CisGRControl
What are our questions?
Is the time to tumor initiation different across treatment groups? is onset later in the cisplatin groups than in the other groups?
Is the growth rate different across treatment groups? is the growth rate for high cisplatin smaller or larger than for low
cisplatin in the GR group?
These are questions that can be addressed statistically.
Vague questions: Is tumor size different? Which treatment is the most effective?
(when?)(using what metric?)
Time to tumor initiation: antitumor effects of GR32191 and cisplatin treatment in immunocompromised mice. Subcutaneous tumors from TCC-SUP human bladder cancer cells were treated with vehicle control (12 mice), GR32191 (15 mice), 5 mg/Kg cisplatin (cisplatin high; 12 mice), 5 mg/kg cisplatin in combination with GR32191 (13 mice), or 0.5 mg/kg (single low-dose cisplatin) alone (10 mice), or 0.5 mg/kg cisplatin in combination with GR32191 (10 mice). Tumor size was measured over time. Kaplan-Meier curves showing time to tumor onset across the treatment groups.
0 20 40 60 80
0.0
0.2
0.4
0.6
0.8
1.0
0 10 20 30 40 50 60 70 80
Pro
po
rtio
n T
um
or-
Fre
e
Time to Tumor (Days)
High Cis+GRHigh CisLow Cis+GRLow CisGRnoGR
Comparison Hazard Ratio
p-value 95% Confidence interval for hazard ratio
No GR vs. GR 2.42 0.02 1.12, 5.22No GR vs. Low 1.24 0.63 0.51, 2.99
No GR vs. Low + GR 29.1 <0.0001 8.70, 97.2No GR vs. High 304.1 <0.0001 65.4, 1414.2
No GR vs. High + GR 556.5 <0.0001 113.7, 2722.7GR vs. Low 0.51 0.12 0.22, 1.20
GR vs. Low + GR 12.0 0.0001 3.29, 43.79GR vs. High 125.6 <0.0001 25.2 626.4
GR vs. High + GR 229.8 <0.0001 43.9, 1203.3Low vs. Low + GR 23.5 <0.0001 5.88, 93.9
Low vs. High 245.7 <0.0001 45.7, 1320.5Low vs. High + GR 449.6 <0.0001 79.8, 2531.5Low + GR vs. High 10.5 <0.0001 3.36, 32.6
Low + GR vs. High + GR 19.1 <0.0001 5.77, 63.4High vs. High + GR 1.83 0.19 0.73, 4.61
Table 2: Hazard ratios comparing time to tumor in treatment groups. A hazard ratio greater than 1.00 implies that the first treatment has shorter time to tumor than the second in the comparison. For example, the hazard ratio comparing Low Cis vs. Low Cis + GR is 23.5. This implies that, at any given time for mice who haven’t yet developed a tumor, mice treated with Low Cis were 23 times more likely to have tumor incidence than mice treated with Low Cisp + GR. Hazard ratios less than 1 imply a protective effect. For example, the hazard ratio for GR vs. Low is 0.51. This implies that for mice who haven’t yet developed a tumor, those treated with GR are 0.51 times as likely to have tumor incidence as mice treated on Low Cis at any given point in time.
0 20 40 60 80
Time (days)
Vo
lum
e
0
30100
300
600
1000
2000
3000
4000 High Cis+GRHigh CisLow Cis+GRLow CisGRControl
Tumor growth rate: how to compare?
0 20 40 60 80
Time from Tumor Initiation (days) or Day 60
Vo
lum
e
0
30100
300
600
1000
2000
3000
4000 High Cis+GRHigh CisLow Cis+GRLow CisGRControl
Data used for tumor growth analysis: notice that for miceWith no tumor onset, they are included from day 60+. RemainingMice all have data shown for volumes>0.
0 20 40 60 80
Time from Injection (days)
Tu
mo
r V
olu
me
High Cis+GRHigh CisLow Cis+GRLow CisGRControl
0
30100
300
600
1000
2000
3000
4000
Fitted regression lines per mouse, by treatment group (stage 1 ofTwo stage analysis).
0 20 40 60 80
Time from Injection (days)
Tu
mo
r V
olu
me
High Cis+GRHigh CisLow Cis+GRLow CisGRControl
0
30100
300
600
1000
2000
3000
4000
Estimated regression lines per treatment group (result of stage 2 ofTwo stage analysis)
Comparisons of slopes
GR Low Cis
Low Cis + GR
High Cis High Cis + GR
Control 0.004 0.0001 <0.0001 0.14 0.0006GR 0.75 0.47 0.08 0.89Low Cis 0.58 0.01 0.84Low Cis + GR 0.003 0.49High Cis 0.03
Table 3: P-values for comparing slopes of tumor growth
Simpler ways to deal with the data?
Simple comparisons across groupsExample:
two groups of mice have tail vein injections to establish tumors. they are followed for a fixed amount of time and then are sacrificed.
Question 1: All mice get tumors. you want to compare tumor burden in the two groups of mice. What test would you use to compare them?
a) t-testb) Wilcoxon rank sum testc) Anovad) Fisher’s exact teste) it depends
Simpler ways to deal with the data?
Simple comparisons across groupsExample:
two groups of mice have tail vein injections to establish tumors. they are followed for a fixed amount of time and then are sacrificed.
Question 3: SOME mice get metastases. You want to compare incidence of metastases. What test should you use?
a) Chi-square testb) Fisher’s exact testc) Anovad) Kaplan Meiere) Signed rank testf) it depends
Simpler ways to deal with the data?
Simple comparisons across groupsExample:
two groups of mice have tail vein injections to establish tumors. they are followed for a fixed amount of time and then are sacrificed.
Question 2: All mice get tumors. you want to compare tumor burden in the two groups of mice. What are you comparing if you use a t-test?
a) the distribution of tumor volumeb) the distribution of the log of tumor volumec) the mean of tumor volume d) the mean of log tumor volumee) It depends
Simpler way to deal with the data?
By using simple approaches, you might be oversimplifying
This can hurt you. Example: you average triplicate values
naively assuming that there is 1/3 of the data as truly exists this will make your standard errors larger than they should be
This can invalidate your results.Example: repeated measures on the same mouse
assumed to be independent naively assuming that there is more data than truly exists. this will make your standard errors smaller than they should
be
5. Lastly…how many mice should you use?
Stay tuned for next week’s talk by EKG.
Quick aside….interpreting pvalues
Definition: the p-value is the probability of getting a result as or more extreme than you observed if the null hypothesis is true
Can anyone translate that?
Hypotheses
Ho: mean gene expression is the same in two groups
H1: mean gene expression is different in two groups
Next: we do an experiment, we collect data (e.g. gene expression), we perform a test.
What will affect the p-value? which hypothesis is actually true the variance of the values in each group the SAMPLE SIZE!!!
What if…?
What if we find a p-value of 0.02 comparing the mean gene expression in two groups? Your conclusion is…. i need more information including:
the effect size! how different is the gene expression? the sample size! how many samples/animals were in
each group? And, it would be nice to ‘see’ the data either in a
figure of by knowing the means and standard deviations in the two groups
Two scenarios
Scenario 1: 5 mice per group. Gene expression is 4 times higher on average in the KO group compared to the WT group. P-value is p=0.02.
Scenario 2: Tissue microarray. Gene expression is 1.2 times higher in late stage cancers (n=100) compared to the normals (n=50) (p=0.02). There is no significant difference (fold change = 1.05) between early stage cancers (n=100) and normals (p=0.45).
Take home points
1. Never interpret a p-value without additional information
2. statisticians can help you. at a minimum, we help you clarify your experimental design and definition of outcomes.
3. statisticians have many tools in our toolboxes. A t-test is not the hammer and every experiment is not a nail.
4. sample size justification will be next….very important (maybe most important) piece of statistical considerations in your grant.