Statistical Design for Biospecimen Cohort Size in Proteomics-based

Statistical Design for Biospecimen Cohort Size in Proteomics-basedBiomarker Discovery and Verication StudiesSteven J. Skates,*, Michael A. Gillette, Joshua LaBaer, Steven A. Carr, Leigh Anderson,,

Daniel C. Liebler, David Ransoho, Nader Rifai,,, Marina Kondratovich, Zivana Tezak,

Elizabeth Manseld, Ann L. Oberg, Ian Wright, Grady Barnes, Mitchell Gail, Mehdi Mesri,

Christopher R. Kinsinger, Henry Rodriguez, and Emily S. Boja*,

Biostatistics Center, Massachusetts General Hospital Cancer Center, Boston, Massachusetts 02114, United StatesProteomics Platform, Broad Institute of Massachusetts Institute of Technology and Harvard, Cambridge, Massachusetts 02142,United States

Personalized Diagnostics, The Biodesign Institute, Arizona State University, Tempe, Arizona 85287, United StatesThe Plasma Proteome Institute, Washington, D.C. 20009, United StatesSISCAPA Technologies, Inc., Washington, D.C. 20009, United StatesJim Ayers Institute for Precancer Detection and Diagnosis and Department of Biochemistry, Vanderbilt University, School ofMedicine, Nashville, Tennessee 37232, United StatesDivision of Gastroenterology and Hepatology, Department of Medicine, University of North Carolina, Chapel Hill, North Carolina27514, United States

Department of Laboratory Medicine, Childrens Hospital Boston, Boston, Massachusetts 02115, United StatesDepartment of Pathology, Harvard Medical School, Boston, Massachusetts 02115, United StatesKing Abdulaziz University, Jeddah 22254, Saudi ArabiaOce of In Vitro Diagnostics and Radiological Health, Center for Devices and Radiological Health, Food and Drug Administration,Department of Health and Human Services, Silver Spring, Maryland 20993, United StatesDivision of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota55905, United StatesFujirebio Diagnostics, Inc., Malvern, Pennsylvania 19355, United StatesDivision of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Department of Health &Human Services, Bethesda, Maryland 20892, United StatesOce of Cancer Clinical Proteomics Research, Center for Strategic Scientic Initiatives, National Cancer Institute, NationalInstitutes of Health, Department of Health & Human Services, Bethesda, Maryland 20892, United States

*S Supporting Information

ABSTRACT: Protein biomarkers are needed to deepen ourunderstanding of cancer biology and to improve our ability todiagnose, monitor, and treat cancers. Important analytical and clinicalhurdles must be overcome to allow the most promising proteinbiomarker candidates to advance into clinical validation studies.Although contemporary proteomics technologies support themeasurement of large numbers of proteins in individual clinicalspecimens, sample throughput remains comparatively low. Thisproblem is amplied in typical clinical proteomics research studies,which routinely suer from a lack of proper experimental design,resulting in analysis of too few biospecimens to achieve adequatestatistical power at each stage of a biomarker pipeline. To address thiscritical shortcoming, a joint workshop was held by the NationalCancer Institute (NCI), National Heart, Lung, and Blood Institute (NHLBI), and American Association for Clinical Chemistry (AACC)

Received: February 11, 2013Published: September 24, 2013

continued...

Article

pubs.acs.org/jpr

2013 American Chemical Society 5383 dx.doi.org/10.1021/pr400132j | J. Proteome Res. 2013, 12, 53835394

Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

INTRODUCTIONCurrent Challenges in Clinical Proteomics

Through the identication of novel plasma and tissue proteinbiomarkers, clinical proteomics has the potential to enableadvances in multiple clinical challenges in cancer, including theimproved detection of cancers and the prediction of ecacy fortherapeutic treatments. Clinical proteomic studies for diagnosticbiomarker discovery in a multistage biomarker pipeline typicallybegin with the identication and measurement of a large numberof proteins in a set of source biospecimens. Dierentiallyexpressed proteins identied in case and control source samplesform the initial protein candidate list. By conrmation ofdierential expression in clinically useful samples such as bloodor serial measurement of fewer proteins with increasing precisionon greater numbers of samples, candidates can be progressivelycredentialed to yield a select few proteins that may warrantassessment in an often time-consuming and costly large-scaleclinical validation trial.1,2

All stages of the proteomics biomarker pipeline presentchallenges to clinical proteomics. These challenges include (1)measurement standardization and optimization for discovery andverication stages, (2) development of high throughput, highprecision, and low cost per sample assays for clinical validationstage, (3) understanding the analytical and clinical validationdesigns acceptable to the FDA, and (4) statistical experimentaldesign at discovery and verication stages of the pipeline that arepoorly understood in comparison to clinical validation.Insucient attention to statistical design has often resulted inthe analysis of too few samples to achieve any reasonablestatistical power to detect biomarkers. Here we address the lastchallenge with regard to sample sizes for protein discovery andverication and derive statistical and clinical criteria foradvancing a biomarker candidate within the pipeline.In collaboration with other eorts, the NCIs Clinical

Proteomic Technologies for Cancer initiative (NCI-CPTC),launched in 2006, helped address some of these barriers,including standardizing proteomic platform measurements andclarifying FDA requirements for analytical validation3 through ajoint NCIFDA workshop.4 The introduction of vericationbetween discovery and clinical validation stages eciently triagesidentied candidates for further assessment in the clinicalvalidation stage where a handful of veried candidates are quan-titatively measured in thousands of clinical samples reecting thefull spectrum of biomarker heterogeneity in the targetpopulation. Verication bridges discovery and validation usingan intermediate number of candidates, precision of measure-ment, length of time for assay development, and sample through-put. To achieve this, verication relies on analytically robust,targeted proteomic platforms and well-characterized anityreagents (if needed) to measure relative concentrations ofa moderate number of biomarker candidates in a moderatenumber of patient samples.To illustrate an example of such a proteomics biomarker

pipeline, we present a proposed scheme for discovery and

verication of plasma biomarkers for dierential diagnosis ofovarian cancers. In the discovery stage, untargeted massspectrometry analysis (shotgun proteomics) of benign andmalignant ovarian cyst uids will yield a large number ofdierentially expressed candidate proteins from a small numberof case and control patient samples. These proteins are rankedfor dierential expression by a t-test on the log-intensity scale.The top 50 candidates are passed onto the verication stage usingquantitative targeted assays in plasma based on multiple reactionmonitoring mass spectrometry (MRM-MS). This technique,commonly used for small molecule and metabolite analysis inclinical laboratories, has recently been optimized usingproteotypic peptides (unique to each target protein) assurrogates for protein concentration measurement. For someof these candidates with low plasma concentrations (nanomolaror less), MRM-MS assays may require the development of anti-peptide antibodies against selected proteotypic peptides prior toMS analysis on a triple quadrupole mass spectrometer (i.e.,immuno-MRM).5 By MRM-MS, plasma from a moderatenumber of cases with malignant ovarian tumors and controlswith benign ovarian disease is measured for the top 50candidates. Subsequently, a t-test of the log-concentrationranks each candidate, and ve top-ranked, veried candidatesare further passed to the clinical validation stage using highthroughput, reliable assays, such as ELISAs. A caveat for devel-oping ELISAs for verication is their higher cost and lead time(>12 months) compared with immuno-MRMs (3 months).Thousands of validation samples are measured for these top veprotein targets in plasma from cases and controls obtained in anunbiased way by drawing blood prior to diagnosis. This sampleset may be split randomly into a training set and a validation set.A classier using the ve best candidates is trained on the trainingset and unbiased operating characteristics are estimated on thevalidation set. If the classiers characteristics reach clinical utility,the biomarker pipeline has identied a biomarker classier. Aforebear of this example pipeline produced OVA1,6 while a moredistant forebear produced ROMA;7 both are FDA authorizedtests for dierential diagnosis of pelvic masses.Statistical Design for a Multistage Proteomics BiomarkerPipeline

Many publications have addressed experimental design of clinicalvalidation studies, sample size calculations in multistage genome-wide association studies,8,9 diagnostic studies in radiology, orcomparison of medical tests.1013 However, sample sizes for amultistage proteomics biomarker pipeline, such as the exampledescribed above, have not been fully addressed. The workshopentitled Experimental Design Considerations in ResearchStudies Using Proteomic Technologies addressed this chal-lenge. Statistical design at each stage of a proteomics biomarkerpipeline with an intended clinical use of an assay (or medicaldevice) as required by the FDA14 is vital to inform whetherthere is sucient power to justify moving candidates to thenext stage. A case study for the detection of ovarian cancers usinga multistage pipeline was used to illustrate statistical design

with participation from the U.S. Food and Drug Administration (FDA). An important output from the workshop was a statisticalframework for the design of biomarker discovery and verication studies. Herein, we describe the use of quantitative clinicaljudgments to set statistical criteria for clinical relevance and the development of an approach to calculate biospecimen sample size forproteomic studies in discovery and verication stages prior to clinical validation stage. This represents a rst step toward building aconsensus on quantitative criteria for statistical design of proteomics biomarker discovery and verication research.

statistical experiment design, biomarker, proteomics, unbiasedness, power calculationKEYWORDS:

Journal of Proteome Research Article

dx.doi.org/10.1021/pr400132j | J. Proteome Res. 2013, 12, 538353945384

Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

considerations involving MS-based technologies15,16 (Methodsand Results). In addition to sample size, designs for unbiasedselection of patient cohorts17 were proposed at this workshopthat will not be described in this work. Compared with previouspublications, this report is novel in three aspects: (1) it addressessample sizes for the discovery and verication stages of a pro-teomics biomarker pipeline (not clinical validation); (2) themodeling accounts for the marker being shed only by a subset oftumors as would be predicted by tumor heterogeneity and asoccurs with known cancer biomarkers; and (3) the selection ofcandidates for successive stages of development includes clinicalcriteria to assess whether a candidate has a chance to have clinicalutility.

Dening Statistical Design Goals

Guided by the context of an intended clinical use, the aim is toidentify experimental designs that will achieve acceptablestatistical power for a true biomarker with specied character-istics to reach clinical validation stage from discovery andverication stages. Verication in the ovarian case study usesMRM-MS, which has previously demonstrated intra- andinterlaboratory precision of %CV < 20%.18 The statistical designand sample sizes presented here, however, apply broadly tobiomarkers other than proteins and technologies other than MS.For instance, autoantibodies are promising alternative bio-markers, and protein arrays that detect autoantibodies in bloodcan be used in discovery and verication stages to triagecandidates using the approaches described herein.19

Each stage in the pipeline winnows the candidate list. Givencurrent MS instrumentation, the discovery stage may detect100010000 proteins from clinical samples. If each stageachieves a 10-fold winnowing, a three-stage pipeline2022

achieves a combined 1000-fold reduction in the number ofcandidates to approximately 110 validated biomarkers. Giventhe current lack of standardization for statistical design, thisreport focuses on designing discovery and verication stages toachieve a given power for a true biomarker to reach clinicalvalidation stage.The decision on whether a candidate proceeds from discovery

to verication stage depends on multiple factors often specic tothe discovery eort, including throughput of sample processing,and both the biological relevance and the statistical signicanceof the candidate. One additional criterion advocated here but notoften addressed by the community is the potential contributionto a clinical question. If the potential contribution is very small,one might consider not advancing such a candidate to the nextstage, even if it were statistically signicant. In a clinical exampleof early detection of ovarian cancer where acceptable perform-ance is most readily dened by the positive predictive value(PPV), we derive the specicity corresponding to the lower limitacceptable (actionable) for the PPV, and incidence of the disease,assuming a sensitivity of 100%. Candidate sensitivities less than5% will not likely be signicant contributors to the ultimate goalof 100% sensitivity. By applying such clinical quantitative criteriain concert with conventional statistical criteria at each stage ofbiomarker development, we ensure that biomarker candidatesemerging at the end of the pipeline will have a reasonableexpectation of contributing signicantly to the sensitivity of abiomarker panel while retaining clinically actionable specicityand PPV. This approach, rst introduced at this workshop andsubsequently presented at a National Institute of Diabetes andDigestive and Kidney Diseases workshop,23 is greatly expandedin this report. Though introduced in the context of ovarian

cancer detection, it is readily generalized to other diagnosticsituations.

METHODSThe Statistical Model

The statistical model aims to simulate the discovery andverication stages to estimate the probability of detecting onebiomarker whose distribution separates cases and controlsamong the thousands of matrix proteins detectable in the sourcebiospecimens whose distribution does not separate cases fromcontrols. The protein molecules in the sample source (e.g.,plasma or cystic uid) except for the biomarker are referred to asthe background matrix. In addition to the biological variationbetween patients, the measurement processes in discovery andverication stages add analytical variation. For the discoveryphase, the measurement CV increases as the concentrationdecreases from 20% for the most abundant proteins to 60% forthe least abundant detectable proteins. Thus, a model for thenumber of proteins at each decade of concentration is required.Hortin and Anderson previously surveyed the rst four decadesof plasma proteins.24 Because the number of proteinsapproximately doubles with each decade, the extrapolation ofthis trend to eight decades provides the model for thedistribution of protein concentrations for a total of 8000matrix proteins. We modeled the measurement variation bythe CV = 15% + (5 decade), where decade = 1, ..., 8, and theproteins are distributed across the concentration decades asdescribed in Figure 1. The biological CV between patients ismore complicated. High abundance proteins have low biologicalCVs (e.g., 9% for albumin), whereas low abundant proteins havea range of biological CVs. There are plasma proteins with lowconcentration and a tight range of concentrations betweenpeople (e.g., many hormones), while other plasma proteins havelarge variation at low concentrations. To derive a statistical modelfor the biological CV as a function of concentration, we estimatedbiological CVs from a database on normal biological variation forlaboratory tests and their normal ranges.25 The details of the sta-tistical modeling are provided in the Supporting Information butin essence consist of increasing expected CV and increasing var-iation in CV with increasing decade of concentration (Figure 2).The total CV in the simulations due to measurement (e.g.,

50%) and biological variation (e.g., 60%) was then calculated as78% = (50%2 + 60%2)1/2. To rst order, the CV is equivalent tothe standard deviation (SD) on the log-concentration scale. Eachdecade of proteins was simulated with the number of proteinsgiven in Figure 1 (except for the last decade in the gure) for atotal of 7899 plasma proteins. Let Y denote the log-concentrationof a (nonbiomarker) protein in a discovery biospecimen, if itcould be measured without error, let X denote variation in log-concentration due to the measurement process, and let Z denotethe log-concentration of a biomarker that has a dierentdistribution in cases compared with controls. The biologicaldistribution across subjects of log-concentration for each matrixprotein, i = 1, ..., P, in cases (j = 2) and controls (j = 1) and insubjects n = 1, ..., N was modeled by Yijn N(i, i), where i =decade = 1, ..., 8 with a probability of selecting the ith decadeproportional to the number of proteins in the decade (Figure 1)and log(i) N(3.60 0.0757 decade, 0.784 0.596 decade).Measurement variation for discovery stage was modeled by

Xijn N(0, i), where i = 15% + 5 decade, decade = 1, ..., 8.



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

The biological variation of the biomarkers concentrationbetween subjects was modeled by Zjkn pkN( +, ) + (1 pk)N(, ) (j = 2 = case), where biomarker has signal = = s for s =2, 3, 4, or 5 in pk = 10%, 20%, 30%, 50%, and 80% of cases, k = 1, 2,3, 4, or 5, and Zjkn N(, ) (j = 1 in control subjects).The overexpression of a protein in a proportion pk of subjects

is = s, where s = 2, 3, 4, or 5 is the number of SDs separating

the mean of the log-concentration of the biomarker in theoverexpressed cases from mean for the biomarker in the controlsubjects. For proximal uids and tissue, the signal/noise ratio ofthe biomarker is likely to be greater than in a uid more remotefrom the disease such as plasma. Therefore, the separation wasmodeled as s = 2, 3, 4, or 5 for proximal uids and tissue in thediscovery stage and s = 1, 2, 3, or 4 in plasma for the vericationstage. Since it is likely that undiscovered biomarkers have lowconcentration, the average concentration for the biomarker wasmodeled at the lowest decade of concentration. Based onvariations in known plasma biomarkers, the biological CV forthe as-yet-undiscovered cancer biomarker was assumed to beabove average compared with other plasma proteins at the samedecade. Thus, log() was set at 1 SD above its expected value.The measurements on P proteins (P = 7899 discovery) weresimulated in N cases and N control subjects, and the t-statisticcomparing cases to controls was calculated for each of the Pmatrix proteins with log-concentration (Yijn + Xijn) where there isno dierence in the distribution between the cases and controls.However, there will be a few proteins among 7899 proteinswhere the N simulated results in the cases appear to besignicantly elevated compared with those in the controls due tochance. The one (biomarker) protein Z with signal in aproportion p of theN cases and no signal in a proportion (1 p)of N cases will have results also simulated in N cases and Ncontrols where there is likely to be a signicant dierence. If thet-statistic for Z is within the top C1 results for discovery stage orC2 results for verication stage, then the biomarker is passedonto the next stage. C1 is the number of targeted assays (e.g.,immuno-MRMs) that the investigator is planning to develop for

Figure 1. Distribution of proteins in blood (plasma or serum) by concentration decade. This is a discrete version of a triangular distribution of thenumber of plasma proteins with increasing concentration decade (adapted from Horton et al.24). Until a human protein quantitation project iscompleted, the distribution of plasma proteins as a function of concentration below 4 logs of concentration is based on an extrapolation.

Figure 2. Distribution of biological CV by concentration decade. Thebiological CV, denoted by , is plotted against the concentration decadefor the table of blood protein tests in Ricos et al.25 A statistical regressionmodel estimates the increasing expected level (blue line) and increasingvariation (red lines, 1SD and 2SD) for as a function of concentrationdecade on the log scale. The model provides estimates for the variationof plasma proteins across the nine decades of concentration simulatedfor the power calculations.



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

the verication stage (C1 = 20, 50, and 100), and C2 is thenumber of high throughput reliable assays that the investigatorplans to develop for the clinical validation stage (e.g., EISAs).C2 is expected to be proportional to the number of targetedassays (C1) and set at 10% of C1 for this simulation. The processwas repeated 1000 times, and the proportion of times that Z wassuccessfully passed to the next stage was calculated. This estimatewas calculated for each combination of the proportion of casesoverexpressing the biomarker p (10%, 20%, 30%, 50%, 80%)with overexpression s = 2, 3, 4, or 5 SDs (or 1, 2, 3, or, 4 SDsfor plasma) in the cases above the controls, and sample sizes of

n = 10, 25, 50, and 100 in the discovery stage, while n = 25, 50,100, and 250 in the verication stage. The results are the powergiven in Tables 1 and 2 with a minimum Monte Carlo accuracyof 1.5% around 50% and 1% or less when the power is 80% orgreater.Factors aecting required sample size to achieve a specied

power to identify the biomarker include the biomarkers amountof separation between cases and controls (Figure 3) andthe bandwidth of the pipeline. The bandwidth is measured bythe number of targeted assays planned to be developed in theverication stage and the number of high throughput, low

Figure 3. Separation of biomarker distribution between cases shedding the biomarker and controls, crossed with fraction of cases shedding biomarker.This gure is a simulation example provided to biomarker researchers in choosing the expected separation between cases and controls (rows) providedby the target biomarker and the fraction of cases shedding the biomarker (column). These two parameters are instrumental in determining the requirednumber of samples. The biomarker distribution in controls is given by the blue histogram with density represented by the dashed line. Cases are amixture of tumors that shed the biomarker and have a distribution (light red) shifted to the right from the biomarker distribution in controls by 5, 4, 3,and 2 SDs for the 1st, 2nd, 3rd and 4th rows, respectively. The proportion of cases shedding the biomarker changes by column from 80% to 50% to 20%in the 1st, 2nd and 3rd column, respectively. Cases that do not shed the biomarker have the same biomarker distribution as controls. The red histogramrepresents the mixture of the cases shedding the biomarker (solid line on right, light red) and the cases not shedding the biomarker (solid line on leftunder the dashed line, dark red). The top left corner (5 SDs of separation with 80% cases shedding biomarker) illustrates the most extreme and easy-to-discover tumor biomarker (CA125). Hence, this situation forms the extreme of the spectrum of separation and fraction of cases shedding the biomarkerwith subsequent examples of decreasing the separation or the fraction shedding the biomarker or both. Biomarker discoverers need to judge where theto-be-discovered biomarker lies within this spectrum and obtain an estimate as to sample size in the discovery and verication stages of a multistageproteomic pipeline.



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

measurement CV assays to be developed in the clinical validationstage. The number of false positive candidate proteins will aectpower and increase as the analytical depth of the discoveryinstrument increases. Finally, between subject variation inprotein log-concentration and variation due to the measure-ment process (analytical variation) increase the required samplesizes (Figure 4). These factors are discussed more fully in theSupporting Information.

Dening Quantitative Criteria Based on Clinical Judgments

In addition to requiring candidates to meet statistical criteria, abiomarker should also have clinical utility. One reasonableapproach to dene quantitative criteria based on clinicalrequirements for candidate biomarkers is as follows:

1. Dene the intended clinical use, including a fulldescription of the clinical management pathway withinwhich the diagnostic test will be applied.

2. Calculate an acceptable benets-to-harms (B/H) ratio, that is,benet of true positive and true negative compared with theharm resulting from false positive and false negative tests.27

3. Derive the required specicity assuming 100% sensitivity toachieve an acceptable B/H ratio within the clinical pathway.

4. Evaluate the sensitivity of biomarker at the requiredspecicity. This sensitivity is likely an upper limit on thecontribution to a panel that aims at 100% sensitivity at therequired specicity.

The rst two steps apply generally, while the third and fourthsteps focus on specicity because in the context of early detectionfor ovarian cancer, harm occurs mostly when false positive resultslead to unnecessary surgery. For intended clinical uses where thegreatest harm occurs due to false negative results, step threeshould focus on sensitivity and step four on evaluating specicityat the required sensitivity. This general approach is approximateand appropriate when disease incidence is low, such as for theearly detection of cancer. The Supporting Information providesthe exact formulas when disease incidence is not low, suchas dierential diagnosis of pelvic masses where incidence of

malignancy is 20%. Harm due to false negatives will be negligibleif the ovarian cancers missed by screening are still detectedclinically due to symptoms at the same time as if screening didnot occur. By benets and harms, we implicitly mean the netbenets and net harms, which assess the dierence betweenoutcomes using the diagnostic test to outcomes under usual carewithout the diagnostic test.Dening the minimally acceptable B/H ratio draws a

somewhat arbitrary boundary, namely, the maximum numberof major harms that are clinically acceptable for each majorbenet provided by the test. Although specics can bechallenging to rigorously defend, such a line in the sand haspreviously been drawn for the early detection of ovarian cancers,where the minimally acceptable B/H ratio was determined to beno more than 10 surgeries to nd 1 screen-detected ovariancancer.28 Setting the discussion within the context of clinical B/Hratio as a formative criterion has the distinct advantage of using ascale on which physicians, patients, and payers can collaborate toform judgments informed by their respective tolerance of riskand benet. Scales such as fold increase in average biomarkerlevel from controls to cases (as commonly used in proteomics) orspecicity or positive likelihood ratio can subsequently bederived from this B/H ratio. In contrast, setting seemingly highbut arbitrary fold increases or specicities creates diculties formost physicians, patients, and payers in forming judgments as towhat is clinically relevant and acceptable. For example, while 85%specicity may be adequate for one intended use, the specicitymay need to be 95% or 98% or 99.8% to achieve clinical utilityin the context of other intended uses, even with a clinicallyacceptable level of sensitivity.For the initial discovery stage, it is reasonable for the B/H

judgments to consider only major eects and for the clinicallymotivated criteria to be estimated to rst order with renementoccurring in parallel with each stage of the pipeline. If thesecriteria are developed at the outset and systematically appliedthroughout the biomarker pipeline, resulting candidates willhave a better chance of meeting the clinical criteria for theirintended use.For ovarian cancers, combined surgery and chemotherapy can

be curative if disease is detected early. However, disease usuallypresents when a patient becomes symptomatic, most often at latestage. Currently, although multiple early detection ovariancancer trials are underway, no method has yet been shown tosignicantly reduce either the number of cancers diagnosedat late rather than early stage or mortality. Based on theaforementioned approach (steps 14), quantitative criteria foran early detection ovarian cancer biomarker or a panel are set asfollows:

1. The intended clinical use is the early detection of ovariancancers, where a clinical management pathway26,29 is

a. annual blood test on all postmenopausal womenb. ultrasound for women with positive blood testc. referral for surgery for women with positive

ultrasound2. The benet of a true positive is the earlier detection of an

ovarian cancer, while the benet of a true negative isreassurance (small). The harm of false positive results issurgery on women with no ovarian cancers, whereas theharm of a false negative is to miss ovarian cancers. In thiscase, we do not consider a volunteers experience of anultrasound when the woman does not have the disease amajor harm compared with unnecessary surgery. With a

Figure 4.Distribution and separation of cases versus controls of CA125in blood. For CA125, the typical median measurement in controls is15 U/mL, while the typical median measurement is 100 U/mL in casesat diagnosis of late stage disease, providing a 6-fold increase or anincrease of 1.9 = log(100/15) on the log scale. With CA125 having aninterperson SD of 0.50 (CV of 50%), this dierence corresponds to asignal of 3.8 SDs. However, CA125 for ovarian cancers is one rareexception where its separation and ubiquity of expression enable it to bedetected with relatively small sample sizes. The detection of otherprotein biomarker candidates would likely require an examination of theimpact of sample sizes on discovery and verication of a signal rangingfrom 1, 2, 3, 4, and 5 SDs.



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

false negative blood test, an ovarian cancer will most likelybe detected with usual clinical care at the same point intime as when no screening is performed; therefore, thenet harm of a false negative is likely to be small comparedwith no screening. Taking account of only the main contri-butions, the B/H ratio is the early detection of ovariancancers compared with the surgeries performed as a resultof false positive tests. Hence, the acceptable B/H ratiois established by determining the number of surgeriesrequired to detect one case of ovarian cancer, that is, a PPVof 20%.26,29

3. The required specicity, derived by assuming 100%sensitivity, is determined through the following consid-erations: Since the annual incidence of ovarian cancers inpostmenopausal population in the U.S. is 1 in 2500, ascreening process needs to increase the incidence inthe test-positive population by 500-fold to achieve the goalof 5 surgeries for 1 ovarian cancer. In screening trials,ultrasound as a second line test reduces false positives by10-fold; thus a plasma biomarker test needs to reduce falsepositives by 50-fold to achieve an overall reduction of 500-fold. This makes the reasonable assumption that a falsepositive ultrasound is statistically independent of a falsepositive blood test. Hence, the required specicity(ReqSP) for the blood test is 98%, which achieves thetarget false positive rate of 1 in 50 (2%). A candidate for anovarian cancer blood test with less than 5% sensitivity at98% specicity at either the transition from discovery stageor verication stage would likely not be continued withinthe pipeline.

RESULTSDiscovery Stage Power

The two probabilities for discovery and verication areindependent of each other due to separate sets of samples usedin these two stages. In order to achieve an overall probability ofdetection of 0.8, the probability of success in each stage needs tobe 0.9 (0.9 0.9 = 0.81). For signals where the number of SDsseparating cases producing the biomarker from controls rangesacross = 2, 3, 4, and 5, the proportion p of cases producing themarker of ranges across 10%, 20%, 30%, 50%, and 80%, and thenumber of candidates being passed from discovery to vericationstage ranges across C1 = 20, 50, and 100, the probability ofsuccess in discovery stage has been estimated through simulationand summarized in Table 1 (n1 = number of cases = number ofcontrols = 10, 25, and 50). For example, with only 10 cases and10 controls and 20 candidates planned for verication, a bio-marker must be expressed in at least 80% of the cases, and casesand controls must be separated by at least 5 SDs to achievegreater than 80% probability of reaching verication. If 50targeted assays are planned for verication, then the power ex-ceeds 90% with the same parameters. As the number of bio-specimens increases, the requirement for the proportion of casesshedding the biomarker or the separation in SDs can relax(decrease) while retaining comparable power. When 25 casesand 25 controls are analyzed, a marker can be expressed in only50% of cases at 2 SDs to achieve an 80% probability of passingthrough discovery to verication (Table 1). An increase ofdiscovery sample sets to 50 cases and 50 controls substantiallyincreases the probability of passing the marker to vericationstage, even if the marker is present in as few as 30% of cases.

Verication Stage Power

Due to generally lower measurement CV of targeted proteomicsapproaches used at the verication stage, with all otherparameters being equal, verication will yield a greater chancethan discovery of identifying the true biomarker. The morecandidates assessed at the verication stage, the greater thechance of passing the biomarker onto the clinical validation stage.The simulations quantify the chance of a biomarker beingveried, given quantication of the above parameters and thenumber of nonbiomarker candidates (false positives) assessed.The probability of verication for a true biomarker has beenestimated when measurement processes have a CV = 15%, withthe number of cases equal to the number of controls and rangingfrom 25, to 50, 100, and 250, and the signal ( SDs) andproportion of cases producing the biomarker p remaining aspreviously dened. For signals = 1, 2, 3, and 4 SDs of the log-scale mean for cases that produce the biomarker above thecontrol mean, p at 10%, 20%, 30%, 50% and 80%, and thenumber of candidates being passed to clinical validation stage formeasurement by targeted high throughput plasma assays (e.g.,ELISAs), C2 = 10% of C1 (2, 5, 10), the probability of successin the verication stage is provided in Table 2 (n2 = number ofcases = number of controls = 25, 50, 100, and 250).The determination of overall sample size and power can be

accomplished by the product of probabilities from both discoveryand verication stages. For example, a biomarker pipeline has93% (= 93% 100%) power, if the marker has a signal = 3.0 SDsin proximal uids with 50% of cases expressing the marker, asample size of 50 cases and 50 controls in discovery stage, and 20candidates to be passed on for verication, followed by a reducedsignal of 2 SDs separating cases from controls in plasma atverication stage with a sample size of n = 100 cases and n = 100controls.

DISCUSSIONWhile previous eorts and publications have discussed the topicof statistical design in omics studies in general [ref 23, http://iom.edu/Reports/2012/Evolution-of-Translational-Omics.aspx], this manuscript introduces a statistical model applicable tothe NCI-CPTAC or other multistage proteomics pipelinesspecically for sample size calculation to achieve a reasonableprobability of successfully passing a biomarker through discoveryand verication to clinical validation. An 80% probability ofreaching the clinical validation stage is attained by setting samplesizes to achieve 90% power for discovery and 90% power forverication stages. Factors that aect the required sample sizeinclude the separation between cases and controls, the fraction ofcases producing the biomarker, and the number of targetedassays to be developed for verication and clinical validationstages. Moreover, we advocate the employment of quantitativecriteria at each stage of the biomarker pipeline based on theclinical B/H ratio for the intended use of a diagnostic test. Thegoal of this approach is to judge whether it may be possible forthe candidate to provide clinical utility (with a recommendationto discard candidates that do not pass this test).Our statistical model takes multiple factors into account,

including the distribution of plasma protein concentrationsacross a wide dynamic range, between person biological varia-tion, measurement variation, etc. The distribution of plasma pro-teins between subjects based on results reported in the literaturefor the rst 4 orders of magnitude of abundance in plasmaprotein concentrations has been published.24 Extrapolating



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

this model across a greater dynamic range provides the basis forsimulating the distribution of low abundance plasma proteins.Due to their multiplicity, some proteins will appear to havesignicantly dierent distributions between the sample of casesand the sample of controls by chance, giving rise to false positiveresults within the simulations. For a biomarker with specied

separation and proportion of cases producing it in excess, these falsepositive results can be overcome with (1) a greater number oftargeted assays for either verication or validation, thereby increasingthe chances the biomarker is passed to the next stage, or (2) a greaternumber of case and control samples, thereby reducing the apparentsignal in the spuriously identied candidates due to their multiplicity.

Table 1. Statistical Power for (a) 20, (b) 50, and (c) 100 Verication Stage Assays Planned



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

The properties of known clinical cancer biomarkers provideguidance on reasonable parameter ranges to expect for the bio-marker to be discovered, such as the separation in SDs betweencases and controls and the proportion of cases for which a bio-marker is overexpressed. Since known clinical cancer biomarkers

are likely to have been detected because of their sucientlystrong signals (separation, % shed), it is assumed that novelcancer biomarkers will likely have less strong signals. Therefore,the parameters from known cancer biomarkers form upperbounds on the parameters for novel cancer biomarkers. With

Table 2. Statistical Power with (a) 20, (b) 50, and (c) 100 Plasma Assays for Verication



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

these models, we have simulated discovery and verication stagesof a biomarker development pipeline, providing estimates of theprobability of detecting a biomarker with a given set of propertiesas a function of sample sizes in each of the two stages.If a true plasma biomarker exists with a specied separation of

cases from controls, the modeling results show the strong eectof the number of clinical specimens on the probability ofsuccessfully passing the biomarker from discovery to vericationstages, as shown in Table 1. For discovery, a sample size of n1 = 50cases and 50 controls is required to have a high chance ofdetecting a (true) biomarker with a reasonable combination ofattributes, expressed in a moderate fraction of cases (>30%) andseparated by 3 SDs in proximal uids. The expectation is thatnew plasma ovarian cancer biomarkers will be expressed in asubset of cases ranging from 25% to 50% and that the separationbetween cases and controls will be modest, on the order of 23SDs for the more highly expressed biomarkers and 3 SDs forthe less expressed biomarkers (e.g., 30% of cases). Under theseplausible circumstances, having a high chance of passing a bio-marker from discovery to verication requires a sample size of 50cases and 50 controls at discovery stage.For true biomarkers separating cases from controls, the chance

that they pass through verication to clinical validation stages ishigh (>90%) for most combinations of separation (SDs betweencases and controls) and fraction of cases producing the bio-marker if the sample size is 250 cases vs 250 controls or someadditional combinations (Table 2). With the sample size of 250/250, the chance of passing through is only

Network (U01CA152990 SJS), and Mayo Clinic SpecializedProgram in Research Excellence grant (P50CA136393 AO).

ABBREVIATIONSNCI-CPTC, NCIs Clinical Proteomic Technologies for Cancerinitiative; CPTAC, Clinical Proteomic Tumor Analysis Con-sortium; NHLBI, National Heart, Lung, and Blood Institute;AACC, American Association of Clinical Chemistry; FDA, USFood and Drug Administration; ESI, electrospray ionization;MRM-MS, multiple reaction monitoring mass spectrometry;CID, collisionally induced dissociation; SISCAPA, stable isotopestandard capture with anti-peptide antibodies; B/H, benets-to-harms ratio; ELISA, enzyme-linked immunosorbent assay; IVD,in vitro diagnostics; RUO, research use only; IUO, investigationaluse only; PTMs, post-translational modications; CLIA, clinicallaboratory improvement amendments; SOP, standard operatingprocedures; SD, standard deviation; CV, coecient of variation;ReqSP, required specicity; PPV, positive predictive value

REFERENCES(1) Rifai, N.; Gillette, M. A.; Carr, S. A. Protein biomarker discoveryand validation: The long and uncertain path to clinical utility. Nat.Biotechnol. 2006, 24 (8), 971983.(2) Boja, E.; Rivers, R.; Kinsinger, C.; Mesri, M.; Hiltke, T.; Rahbar, A.;Rodriguez, H. Restructuring proteomics through verification. Bio-markers Med. 2010, 4 (6), 799803.(3) Regnier, F. E.; Skates, S. J.; Mesri, M.; Rodriguez, H.; Tezak, Z.;Kondratovich, M. V.; Alterman, M. A.; Levin, J. D.; Roscoe, D.; Reilly,E.; Callaghan, J.; Kelm, K.; Brown, D.; Philip, R.; Carr, S. A.; Liebler, D.C.; Fisher, S. J.; Tempst, P.; Hiltke, T.; Kessler, L. G.; Kinsinger, C. R.;Ransohoff, D. F.; Mansfield, E.; Anderson, N. L. Protein-BasedMultiplex Assays: Mock Presubmissions to the US Food and DrugAdministration. Clin. Chem. 2010, 56 (2), 165171.(4) Rodriguez, H.; Tezak, Z.; Mesri, M.; Carr, S. A.; Liebler, D. C.;Fisher, S. J.; Tempst, P.; Hiltke, T.; Kessler, L. G.; Kinsinger, C. R.;Philip, R.; Ransohoff, D. F.; Skates, S. J.; Regnier, F. E.; Anderson, N. L.;Mansfield, E. Workshop Participants. Analytical Validation of Protein-Based Multiplex Assays: A Workshop Report by the NCI-FDAInteragency Oncology Task Force on Molecular Diagnostics. Clin.Chem. 2010, 56 (2), 237243.(5) Whiteaker, J. R.; Zhao, L.; Anderson, L.; Paulovich, A. G. Anautomated and multiplexed method for high throughput peptideimmunoaffinity enrichment and multiple reaction monitoring massspectrometry-based quantification of protein biomarkers. Mol. Cell.Proteomics 2010, 9 (1), 184196.(6) Zhang, Z.; Chan, D. W. The road from discovery to clinicaldiagnostics: Lessons learned from the first FDA-cleared in vitrodiagnostic multivariate index assay of proteomic biomarkers. CancerEpidemiol., Biomarkers Prev. 2010, 19 (12), 29952999.(7) Bast, R. C., Jr.; Skates, S.; Lokshin, A.; Moore, R. G. Differentialdiagnosis of a pelvic mass: improved algorithms and novel biomarkers.Int. J. Gynecol. Cancer 2012, 22 (Suppl 1), S58.(8) Gail, M. H.; Pfeiffer, R. M.; Wheeler, W.; Pee, D. Probability that atwo-stage genome-wide association study will detect a disease-associated SNP and implications for multistage designs. Ann. Hum.Genet. 2008, 72 (Pt 6), 812820.(9) Satagopan, J. M.; Venkatraman, E. S.; Begg, C. B. Two-stagedesigns for gene-disease association studies with sample size constraints.Biometrics 2004, 60 (3), 589597.(10) Obuchowski, N. A.; Hillis, S. L. Sample size tables for computer-aided detection studies. AJR, Am. J. Roentgenol. 2011, 197 (5), W821828.(11) Obuchowski, N. A.; Zhou, X. H. Prospective studies of diagnostictest accuracy when disease prevalence is low. Biostatistics 2002, 3 (4),477492.(12) Obuchowski, N. A. Sample size tables for receiver operatingcharacteristic studies. AJR, Am. J. Roentgenol. 2000, 175 (3), 603608.

(13) Alonzo, T. A.; Pepe, M. S.; Moskowitz, C. S. Sample sizecalculations for comparative studies of medical tests for detectingpresence of disease. Stat. Med. 2002, 21 (6), 835852.(14) Boja, E. S.; Jortani, S. A.; Ritchie, J.; Hoofnagle, A. N.; Tezak, Z.;Mansfield, E.; Keller, P.; Rivers, R. C.; Rahbar, A.; Anderson, N. L.;Srinivas, P.; Rodriguez, H. The journey to regulation of protein-basedmultiplex quantitative assays. Clin. Chem. 2011, 57 (4), 560567.(15) Rodriguez, H.; Rivers, R.; Kinsinger, C.; Mesri, M.; Hiltke, T.;Rahbar, A.; Boja, E. S. Reconstructing the pipeline by introducingmultiplexed multiple reaction monitoring mass spectrometry for cancerbiomarker verification: an NCI-CPTC initiative perspective. ProteomicsClin Appl. 2010, 4 (12), 904914.(16) Witkowska, H. E.; Hall, S. C.; Fisher, S. J. Breaking the bottleneckin the protein biomarker pipeline. Clin. Chem. 2012, 58 (2), 321323.(17) Ransohoff, D. How to improve reliability and efficiency ofresearch about molecular markers: roles of phases, guidelines, and studydesign. J. Clin. Epidemiol. 2007, 60 (12), 12051219.(18) Addona, T. A.; Abbatiello, S. E.; Schilling, B.; Skates, S. J.;Mani, D.R.; Bunk, D. M.; Spiegelman, C. H.; Zimmerman, L. J.; Ham, A. J.;Keshishian, H.; Hall, S. C.; Allen, S.; Blackman, R. K.; Borchers, C. H.;Buck, C.; Cardasis, H. L.; Cusack, M. P.; Dodder, N. G.; Gibson, B. W.;Held, J. M.; Hiltke, T.; Jackson, A.; Johansen, E. B.; Kinsinger, C. R.; Li,J.; Mesri, M.; Neubert, T. A.; Niles, R. K.; Pulsipher, T. C.; Ransohoff,D.; Rodriguez, H.; Rudnick, P. A.; Smith, D.; Tabb, D. L.; Tegeler, T. J.;Variyath, A. M.; Vega-Montoto, L. J.; Wahlander, A.; Waldemarson, S.;Wang, M.; Whiteaker, J. R.; Zhao, L.; Anderson, N. L.; Fisher, S. J.;Liebler, D. C.; Paulovich, A. G.; Regnier, F. E.; Tempst, P.; Carr, S. A. AMulti-site assessment of precision and reproducibility of multiplereaction monitoring-based measurements by the NCI-CPTAC net-work: toward quantitative protein biomarker verification in humanplasma. Nat. Biotechnol. 2009, 27 (7), 633641.(19)Wallstrom, G.; Anderson, K. S.; LaBaer, J. Biomarker discovery forheterogeneous diseases. Cancer Epidemiol., Biomarkers Prev. 2013, 22(5), 747755.(20) Addona, T. A.; Shi, X.; Keshishian, H.; Mani, D. R.; Burgess, M.;Gillette, M. A.; Clauser, K. R.; Shen, D.; Lewis, G. D.; Farrell, L. A.; Fifer,M. A.; Sabatine, M. S.; Gerszten, R. E.; Carr, S. A. A pipeline thatintegrates the discovery and verification of plasma protein biomarkersreveals candidate markers for cardiovascular disease. Nat. Biotechnol.2011, 29, 635643.(21)Whiteaker, J. R.; Lin, C.; Kennedy, J.; Hou, L.; Trute, M.; Sokal, I.;Yan, P.; Schoenherr, R.M.; Zhao, L.; Voytovich, U. J.; Kelly-Spratt, K. S.;Krasnoselsky, A.; Gafken, P. R.; Hogan, J. M.; Jones, L. A.; Wang, P.;Amon, L.; Chodosh, L. A.; Nelson, P. S.; McIntosh, M. W.; Kemp, C. J.;Paulovich, A. G. A targeted proteomics-based pipeline for verification ofbiomarkers in plasma. Nat. Biotechnol. 2011, 29 (7), 625634.(22) Gerszten, R. E.; Asnani, A.; Carr, S. A. Status and prospects fordiscovery and verification of new biomarkers of cardiovascular diseaseby proteomics. Circ. Res. 2011, 109 (4), 463474.(23) Vidal, M.; Chan, D. W.; Gerstein, M.; Mann, M.; Omenn, G. S.;Tagle, D.; Sechi, S. Workshop Participants. The human proteome - ascientific opportunity for transforming diagnostics, therapeutics, andhealthcare. Clin. Proteomics 2012, 9 (1), 617.(24) Hortin, G. L.; Sviridov, D.; Anderson, N. L. High-abundancepolypeptides of the human plasma proteome comprising the top 4 logsof polypeptide abundance. Clin. Chem. 2008, 54 (10), 16081616.(25) Ricos, C.; Alvarez, V.; Cava, F.; Garca-Lario, J. V.; Hernandez, A.;Jimenez, C. V.; Minchinela, J.; Perich, C.; Simon, M. Current databaseson biological variation: pros, cons and progress. Scand. J. Clin. Lab. Invest.1999, 59 (7), 491500.(26)Menon, U.; Gentry-Maharaj, A.; Hallett, R.; Ryan, A.; Burnell, M.;Sharma, A.; Lewis, S.; Davies, S.; Philpott, S.; Lopes, A.; Godfrey, K.;Oram, D.; Herod, J.; Williamson, K.; Seif, M. W.; Scott, I.; Mould, T.;Woolas, R.; Murdoch, J.; Dobbs, S.; Amso, N. N.; Leeson, S.;Cruickshank, D.; McGuire, A.; Campbell, S.; Fallowfield, L.; Singh,N.; Dawnay, A.; Skates, S. J.; Parmar, M.; Jacobs, I. Sensitivity andspecificity of multimodal and ultrasound screening for ovarian cancer,and stage distribution of detected cancer: Results of the prevalence



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

screen of the UK Collaborative Trial of Ovarian Cancer Screening(UKCTOCS). Lancet Oncol. 2009, 10 (4), 327340.(27) Jacobs, I.; Bast, R. C., Jr. The CA125 tumour-associated antigen: Areview of the literature. Hum. Reprod. 1989, 4 (1), 112.(28) Altar, C. A.; Amakye, D.; Bounos, D.; Bloom, J.; Clack, G.; Dean,R.; Devanarayan, V.; Fu, D.; Furlong, S.; Hinman, L.; Girman, C.; Lathia,C.; Lesko, L.; Madani, S.; Mayne, J.; Meyer, J.; Raunig, D.; Sager, P.;Williams, S. A.; Wong, P.; Zerba, K. A Prototypical Process for CreatingEvidentiary Standards for Biomarkers and Diagnostics. Clin. Pharmacol.Ther. 2008, 83 (2), 368371.(29) Jacobs, I. J.; Skates, S. J.; MacDonald, N.; Menon, U.; Rosenthal,A. N.; Davies, A. P.; Woolas, R.; Jeyarajah, A. R.; Sibley, K.; Lowe, D. G.;Oram, D. H. Screening for ovarian cancer: A pilot randomizedcontrolled trial. Lancet 1999, 353 (9160), 12071210.



Dow

nloa

ded

by N

ATL

LBR

Y O

F SE

RBIA

on

Sept

embe

r 11,

201

5 | ht

tp://p

ubs.ac

s.org

Pu

blic

atio

n D

ate

(Web

): Oc

tober

28, 2

013 |

doi: 1

0.1021

/pr400

132j

Date post:	10-Jan-2016
Category:	Documents
Upload:	romana-masnikosa
View:	216 times
Download:	0 times

Statistical Design for Biospecimen Cohort Size in Proteomics-based

Documents