Randomized Controlled Trials: Randomized Controlled Trials: Allocation ConcealmentAllocation Concealment

Kenneth F. Schulz, PhDKenneth F. Schulz, PhD

Fixation on Sequence GenerationFixation on Sequence Generation

Traditionally, many medical researchers Traditionally, many medical researchers mistakenly consider the sequence mistakenly consider the sequence generation process as “randomization”generation process as “randomization”

They frequently ignore allocation They frequently ignore allocation concealment concealment

Without adequate allocation concealment, Without adequate allocation concealment, proper random sequences can be subvertedproper random sequences can be subverted

For example, a properly generated sequence For example, a properly generated sequence posted on a bulletin board posted on a bulletin board

Allocation ConcealmentAllocation Concealment

Prevents selection bias by Prevents selection bias by concealing the allocation sequence concealing the allocation sequence from those assigning participants to from those assigning participants to intervention groupsintervention groups

Until the moment of assignmentUntil the moment of assignment

Mechanisms for Selection BiasMechanisms for Selection Biasin RCTsin RCTs

If those responsible for entering If those responsible for entering participants know, or can detect, the participants know, or can detect, the upcoming treatment allocations:upcoming treatment allocations: Decide entrance based on that knowledgeDecide entrance based on that knowledge

Channel participants with a better prognosis Channel participants with a better prognosis to the experimental group and those with a to the experimental group and those with a poorer prognosis to the control group, or vice poorer prognosis to the control group, or vice versaversa

In any case, they introduce bias into the In any case, they introduce bias into the treatment comparisontreatment comparison

Do Not Confuse Allocation Do Not Confuse Allocation ConcealmentConcealment with with BlindingBlinding

Allocation concealment seeks to prevent Allocation concealment seeks to prevent selection bias, protects assignment selection bias, protects assignment sequence before and until allocation, and sequence before and until allocation, and can always be successfully implementedcan always be successfully implemented

In contrast, blinding seeks to prevent In contrast, blinding seeks to prevent ascertainment bias, protects sequence ascertainment bias, protects sequence after allocation, and cannot always be after allocation, and cannot always be successfully implementedsuccessfully implemented

Importance of Allocation Importance of Allocation ConcealmentConcealment

Unclearly concealed and inadequately Unclearly concealed and inadequately concealed trials, compared to concealed trials, compared to adequately concealed trials, adequately concealed trials, exaggeratedexaggerated the estimates of an the estimates of an interventionintervention’’s effectiveness by s effectiveness by 30% to 30% to 40%,40%, on average on average

Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-412.

Similar Results on Allocation Similar Results on Allocation Concealment in a Separate StudyConcealment in a Separate Study

Replicated our methods in different Replicated our methods in different subject areas (digestive & circulatory subject areas (digestive & circulatory diseases and mental health) diseases and mental health)

Examined 11 meta-analysesExamined 11 meta-analyses

Inadequately concealed trials Inadequately concealed trials exaggeratedexaggerated estimates of effectiveness by estimates of effectiveness by 37%37%

[Moher D, et al. Does quality of reports of randomised trials [Moher D, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-affect estimates of intervention efficacy reported in meta-analyses? analyses? LancetLancet 1998; 352: 609-13.] 1998; 352: 609-13.]

Allocation Concealment: SummaryAllocation Concealment: Summary

Adequate allocation concealment emerges Adequate allocation concealment emerges from our analyses as crucial to reducing from our analyses as crucial to reducing biasbias

Without it, the whole point of Without it, the whole point of randomisation vanishes and bias may randomisation vanishes and bias may distort resultsdistort results

Our results support Mosteller and his Our results support Mosteller and his colleaguescolleagues

“When the randomization leaks, the trial’s guarantee of lack of bias runs down the drain”

“When the randomization leaks, the trial’s guarantee of lack of bias runs down the drain”

Method of allocation of treatment in 208 controlled trials in head injury

Dickinson K, et al., BMJ 2000;320:1308-1311.

Method of allocation No. of trials

Method of allocation No. of trialsAdequateAdequate

Centralised randomisation by telephone 1

Numbered/coded identical containers administered sequentially 11

Randomisation scheme controlled by pharmacy 8

Sequentially numbered, sealed, opaque envelopes 2

Not adequateNot adequate

Other 18

Date of birth 1

Day of week 3

Alternation 3

Not stated 161

Actual allocation method with the Actual allocation method with the method just specified as “random”method just specified as “random”

Difficult (Peto and I. Chalmers experience)Difficult (Peto and I. Chalmers experience)

MA of OCsMA of OCs

David G. knew the PI who put him in David G. knew the PI who put him in contact with the statisticiancontact with the statistician

Described a “great” method . . . Post-it Described a “great” method . . . Post-it notes!notes!

Seemed indignant . . . We asked for Seemed indignant . . . We asked for clarificationclarification

Perfect also for stratification—different colors!!Perfect also for stratification—different colors!!

Personal Accounts of Deciphering Personal Accounts of Deciphering Assignment SequencesAssignment Sequences

Conducted over 20 epidemiological Conducted over 20 epidemiological workshops for medical residents and workshops for medical residents and medical junior facultymedical junior faculty

Each included 20-25 participantsEach included 20-25 participants

Asked how many of the participants had Asked how many of the participants had deciphered, or had witnessed someone deciphered, or had witnessed someone else decipher, an assignment sequenceelse decipher, an assignment sequence

With assurance of anonymity, more With assurance of anonymity, more than half related at least one than half related at least one instance of decipheringinstance of deciphering

Not half of all trialsNot half of all trials No accurate denominatorNo accurate denominator

Nevertheless, not a rare occurrenceNevertheless, not a rare occurrence

Personal Accounts of Deciphering Personal Accounts of Deciphering Assignment Sequences Assignment Sequences (cont.)(cont.)

Decipherings Run the Gamut Decipherings Run the Gamut From Simple to IntricateFrom Simple to Intricate

Simple the most frequent and usually Simple the most frequent and usually take advantage of inadequate take advantage of inadequate allocation concealment schemes:allocation concealment schemes: Posting sequence on a bulletin boardPosting sequence on a bulletin board Opening unsealed envelopesOpening unsealed envelopes Translucent envelopes to a light bulbTranslucent envelopes to a light bulb Opening many envelopes that were not Opening many envelopes that were not

sequentially numberedsequentially numbered

Needed to circumvent more adequate Needed to circumvent more adequate allocation schemesallocation schemes With sequentially numbered drug With sequentially numbered drug

containerscontainers Based on appearance of tablets in Based on appearance of tablets in

unsealed containersunsealed containers Appearance of the labelAppearance of the label

With central allocation, obtaining the With central allocation, obtaining the next few allocations at oncenext few allocations at once

Envelopes to a “hot light” in radiologyEnvelopes to a “hot light” in radiology

Intricate Decipherings Less FrequentIntricate Decipherings Less Frequent

Attempted to Decipher a Numbered Attempted to Decipher a Numbered Container SchemeContainer Scheme

Gave upGave up

Found attending physician rifling Found attending physician rifling the P.I.’s files for the assignment the P.I.’s files for the assignment sequencesequence

Horrified? No, impressed with his Horrified? No, impressed with his brilliance and proceeded to helpbrilliance and proceeded to help

Should be kept in a locked locationShould be kept in a locked location

Ignorance Is Not BlissIgnorance Is Not Bliss

““At the annual meeting of the Society for At the annual meeting of the Society for Clinical Trials, held in Houston, … Schulz Clinical Trials, held in Houston, … Schulz reported data showing that the experts’ fears reported data showing that the experts’ fears about ignorance are justified. Schulz about ignorance are justified. Schulz examined 250 reports of clinical trials and examined 250 reports of clinical trials and came up with an intriguing finding…”came up with an intriguing finding…”

““At the annual meeting of the Society for At the annual meeting of the Society for Clinical Trials, held in Houston, … Schulz Clinical Trials, held in Houston, … Schulz reported data showing that the experts’ fears reported data showing that the experts’ fears about ignorance are justified. Schulz about ignorance are justified. Schulz examined 250 reports of clinical trials and examined 250 reports of clinical trials and came up with an intriguing finding…”came up with an intriguing finding…”

[Rachel Nowak. Problems in clinical trials go far beyond misconduct. [Rachel Nowak. Problems in clinical trials go far beyond misconduct. Science 1994; 264: 1539.]Science 1994; 264: 1539.]

Ignorance Is Not BlissIgnorance Is Not Bliss

“Most physicians are not trained in basic scientific principles, let alone clinical trials”

“Most physicians are not trained in basic scientific principles, let alone clinical trials”

““I fantasize that in the future (training) I fantasize that in the future (training) will become a requirement for running will become a requirement for running clinical trials”clinical trials”

““I fantasize that in the future (training) I fantasize that in the future (training) will become a requirement for running will become a requirement for running clinical trials”clinical trials”

John Gallin, Director of NIH’s Clinical CenterJohn Gallin, Director of NIH’s Clinical Center

Stephen George, Duke University, Chair of the Statistics Stephen George, Duke University, Chair of the Statistics Committee for NCI, NIH Cancer Clinical Cooperative GroupsCommittee for NCI, NIH Cancer Clinical Cooperative Groups

Rigorous Trials Annoy HumansRigorous Trials Annoy Humans

InvestigatorsInvestigators Certain Ps to benefitCertain Ps to benefit May want the results of study to May want the results of study to

reveal the “truth”reveal the “truth”

Proper trial procedures attempt Proper trial procedures attempt to impede human inclinationsto impede human inclinations Hence, aspects of properly Hence, aspects of properly

conducted RCTs annoy conducted RCTs annoy investigatorsinvestigators

Without Methodological RigourWithout Methodological Rigour

The challenge of deciphering may frequently The challenge of deciphering may frequently become too great a temptation to resistbecome too great a temptation to resist Even without intent to biasEven without intent to bias

““The only way to get rid of a temptation is to The only way to get rid of a temptation is to yield to it.” yield to it.” Oscar WildeOscar Wilde

Deciphering may just reflect human Deciphering may just reflect human inquisitiveness and ingenuityinquisitiveness and ingenuity

Does not necessarily indicate scientific Does not necessarily indicate scientific malevolencemalevolence

Without Methodological RigourWithout Methodological Rigour

Or deciphering may reflect deliberate Or deciphering may reflect deliberate acts to alter findingsacts to alter findings

Whatever the motivations, innocent Whatever the motivations, innocent or naïve or deliberate, those actions or naïve or deliberate, those actions undermine the validity of the trialundermine the validity of the trial

Investigators must Investigators must devote diligent, devote diligent, persistent attentionpersistent attention to randomization to randomization

Minimal Standards: Common Minimal Standards: Common Allocation Concealment ApproachesAllocation Concealment Approaches

Sequentially numbered, Sequentially numbered, opaque, sealed envelopes opaque, sealed envelopes (SNOSE)(SNOSE)

Pharmacy controlPharmacy control

Numbered or coded containersNumbered or coded containers

Central randomizationCentral randomization

Realistically, these standards Realistically, these standards should be should be exceededexceeded

Envelopes for Allocation Envelopes for Allocation ConcealmentConcealment

More susceptible to More susceptible to manipulation through human manipulation through human ingenuityingenuity

Less than ideal method of Less than ideal method of allocation concealmentallocation concealment

If used, investigators must If used, investigators must diligently develop and monitor diligently develop and monitor the processthe process

Envelopes Envelopes (Cont.)(Cont.)

onlyonly after the P’s after the P’s name and other name and other details are written on details are written on the appropriate the appropriate envelopeenvelope

SNOSESNOSE

Ensure that the envelopes are opened Ensure that the envelopes are opened sequentiallysequentially

Envelopes Envelopes (Cont’d)(Cont’d)

Pressure-sensitive or carbon Pressure-sensitive or carbon paper insidepaper inside Supports enrolment Supports enrolment Encourages orderly openingEncourages orderly opening Promotes proper assignmentPromotes proper assignment Creates valuable audit trailCreates valuable audit trail

Cardboard or aluminum foil Cardboard or aluminum foil inside envelopeinside envelope

““Pharmacy Controlled”Pharmacy Controlled”

Compliance with randomization/allocation Compliance with randomization/allocation concealment methods questionableconcealment methods questionable

Aware of gross distortionsAware of gross distortionsPharmacy ran out of one drug . . .Pharmacy ran out of one drug . . .Alternate assignmentAlternate assignment

Should not assume pharmacists knowledgeShould not assume pharmacists knowledge

Ensure they, and all research partners, Ensure they, and all research partners, follow proper trial proceduresfollow proper trial procedures

““Numbered Containers”Numbered Containers”

Excellent – prevents foreknowledgeExcellent – prevents foreknowledgeMust take proper precautionsMust take proper precautions

Appearance – exactly the sameAppearance – exactly the same

Weight – exactly the sameWeight – exactly the same

Sound when shaken – exactly the sameSound when shaken – exactly the same

Lettering – exactly the sameLettering – exactly the same

Odor – exactly the sameOdor – exactly the same

SealedSealedDifference between concealment and blindingDifference between concealment and blinding

““Central Randomization”Central Randomization”

Frequently telephoneFrequently telephoneFaxFaxEmailEmailInternetInternet““Computer”Computer”

Assignments only provided after Assignments only provided after participants irrevocably enrolled participants irrevocably enrolled

Must check procedures for leaksMust check procedures for leaks Monitor process for adherenceMonitor process for adherence

RCTs: Anathema to the Human SpiritRCTs: Anathema to the Human Spirit

Must acknowledge the Must acknowledge the vagaries of human naturevagaries of human nature

Must establish methodological Must establish methodological safeguards that thwart safeguards that thwart attempts to contaminate trials attempts to contaminate trials with biaswith bias

Subversions of SequencesSubversions of Sequences

Conscious or subconscious?Conscious or subconscious?

Direct or compensatory?Direct or compensatory?

Doesn’t matterDoesn’t matter Biases the trialBiases the trial

ParticipantsParticipants

OutcomeOutcome OutcomeOutcome

PlaceboPlacebo

40% Non-40% Non-compliancecompliance

60% 60% ComplianceCompliance

Group representing the Group representing the policy of no treatmentpolicy of no treatment

25% Non-25% Non-compliancecompliance

75% 75% ComplianceCompliance

Group representing the Group representing the policy of oral treatmentpolicy of oral treatment

New Oral DrugNew Oral Drug

Randomize

RCT Compared the Effectiveness of Clofibrate in RCT Compared the Effectiveness of Clofibrate in Preventing Cardiac Deaths in Men Who Had Preventing Cardiac Deaths in Men Who Had

Survived a Myocardial InfarctionSurvived a Myocardial Infarction

ClofibrateClofibrate PlaceboPlacebo

5 Year mortality5 Year mortality 20.2%20.2% 20.9%20.9% (p = .55)(p = .55)

Authors state that:Authors state that:

• One can justify almost any conclusion, dependent upon the analysis One can justify almost any conclusion, dependent upon the analysis chosenchosen

• Manipulating deviates leads to severe biasManipulating deviates leads to severe bias

• Can you ever do so?Can you ever do so?

Eliminating deviates from Eliminating deviates from clofibrateclofibrate

(80% adherence)(80% adherence)15.0%15.0% 20.9%20.9% (p < .05)(p < .05)

Eliminating deviates from Eliminating deviates from both groupsboth groups 15.0%15.0% 15.1%15.1%

Exclusions After Exclusions After RandomizationRandomization

Can introduce bias and should Can introduce bias and should be carefully scrutinizedbe carefully scrutinized

All randomized patients should All randomized patients should be analyzed, and analyzed as be analyzed, and analyzed as part of the group to which they part of the group to which they were initially assignedwere initially assigned

ITT (Intention-to-treat)ITT (Intention-to-treat)

Exclusions of LFU Damage Exclusions of LFU Damage Internal ValidityInternal Validity

Without outcomes from those lost to Without outcomes from those lost to follow-up, investigators have little choice follow-up, investigators have little choice but to exclude them from the analysis but to exclude them from the analysis

Any losses damage internal validityAny losses damage internal validity

However, differential rates of loss among However, differential rates of loss among comparison groups cause major damage comparison groups cause major damage

Investigators must minimize their losses Investigators must minimize their losses to follow-upto follow-up

Retention of Trial ParticipantsRetention of Trial Participants

Minimizing losses exudes difficultiesMinimizing losses exudes difficulties Eliminating losses may be impossible Eliminating losses may be impossible

But investigators too frequently profess But investigators too frequently profess insurmountable difficultiesinsurmountable difficulties

Most should work harder to obtain Most should work harder to obtain higher follow-up rateshigher follow-up rates

Investigators must commit adequate Investigators must commit adequate attention and resources to develop and attention and resources to develop and implement procedures to minimize implement procedures to minimize losseslosses

Innovative Twists that Cultivate Innovative Twists that Cultivate High Follow-up RatesHigh Follow-up Rates

Establish many conveniently placed FU Establish many conveniently placed FU facilities facilities

Too often investigators expect participants to Too often investigators expect participants to visit a single, inconvenient locationvisit a single, inconvenient location

Shortening data collection form caters to the Shortening data collection form caters to the participants’ wishesparticipants’ wishes

Long sessions lead them to vote with their feetLong sessions lead them to vote with their feetBe considerateBe considerateFoster follow-up by not overburdening Foster follow-up by not overburdening Also may engender higher quality data on theAlso may engender higher quality data on themain itemsmain items

ENDEND

Meta-analysis of polyglycolic acid versus Meta-analysis of polyglycolic acid versus catgut for perineal repair on short term paincatgut for perineal repair on short term pain

1 adequately concealed trial … 1 adequately concealed trial … OR = .89OR = .894 unclearly concealed trials … OR = .444 unclearly concealed trials … OR = .44

Protective effect of polyglycotic acid was 51% lower in the unclearly concealed trialProtective effect of polyglycotic acid was 51% lower in the unclearly concealed trial

(ROR = .49; 95% CI of .35 - .69)(ROR = .49; 95% CI of .35 - .69)

Meta-analysis of PolyglycolicMeta-analysis of Polyglycolic

Meta-analysis of corticosteroids vs. none, after Meta-analysis of corticosteroids vs. none, after preterm rupture of membranes, on respiratory preterm rupture of membranes, on respiratory distress syndrome (RDS)distress syndrome (RDS)

3 adequately concealed trials … OR = 0.723 adequately concealed trials … OR = 0.724 unclearly concealed trials … OR = 0.534 unclearly concealed trials … OR = 0.53

Protective effect of corticosteroids was exaggerated by 27% in the unclearly concealed trials

Protective effect of corticosteroids was exaggerated by 27% in the unclearly concealed trials

ROR = 0.73; 95% CI of 0.35 - 1.50ROR = 0.73; 95% CI of 0.35 - 1.50

Meta-analysis of CorticosteriodsMeta-analysis of Corticosteriods

Randomization Process (P 138B)Randomization Process (P 138B)

In the 4 OB/GYN Journals, found the 2 treatment group sizes to In the 4 OB/GYN Journals, found the 2 treatment group sizes to be much too similarbe much too similar

ExpectedExpected ObservedObserved

5 ( 5%)5 ( 5%) 00

48 (50%)48 (50%) 88

Proper approach to: 4 OB/GYN Journals

4 General Medical Journals

Method of generation of random numbers

32% 49%

Randomization concealment

23% 26%

Both 9% 15%

Separation of Executer and Separation of Executer and GeneratorGenerator

Person(s) who prepare the randomization Person(s) who prepare the randomization scheme should not determine eligibility, scheme should not determine eligibility, administer treatment, or assess outcomeadminister treatment, or assess outcome

Neglected critical elementNeglected critical element

Obviously important—regardless of Obviously important—regardless of methodological quality, access to sequence methodological quality, access to sequence and thus opportunity to introduce biasand thus opportunity to introduce bias

Faults in this element may be the crack Faults in this element may be the crack though which much of the bias seeps into though which much of the bias seeps into controlled trialscontrolled trials

““Not quite as random as I pretended”Not quite as random as I pretended”[Martyn C. Not quite as random as I pretended. Lancet [Martyn C. Not quite as random as I pretended. Lancet

1996; 347:70.]1996; 347:70.]

“Bias in the way treatments are assigned can be a more powerful determinant of the outcome of the trial than the treatments that are being investigated.”

“Bias in the way treatments are assigned can be a more powerful determinant of the outcome of the trial than the treatments that are being investigated.”

““Trialists … often find it hard to prevent Trialists … often find it hard to prevent their hopes that a new treatment might their hopes that a new treatment might prove to be beneficial from evolving intoprove to be beneficial from evolving intoa belief that it really is so.”a belief that it really is so.”

““Trialists … often find it hard to prevent Trialists … often find it hard to prevent their hopes that a new treatment might their hopes that a new treatment might prove to be beneficial from evolving intoprove to be beneficial from evolving intoa belief that it really is so.”a belief that it really is so.”

RCT in NorwayRCT in Norway

Examined the effect of Chlorhexidine Examined the effect of Chlorhexidine compared to placebo on postpartum compared to placebo on postpartum infection and Strep B infections in infection and Strep B infections in neonatesneonates

Reported as a randomized controlled Reported as a randomized controlled trial in the published articletrial in the published article

Insert published article

RCT in NorwayRCT in Norway

Midwives knew on which shelf the Midwives knew on which shelf the Chlorhexidine and placebo were kept in Chlorhexidine and placebo were kept in the cabinet and just proceeded to the cabinet and just proceeded to administer whatever they deemed administer whatever they deemed appropriateappropriate

Even if they forgot the shelf, they could Even if they forgot the shelf, they could smell the differencesmell the difference

No monitoring of the allocation processNo monitoring of the allocation process

Letter from a Physical TherapistLetter from a Physical Therapist

A A “friend of mine”“friend of mine” conducted an RCT conducted an RCT comparing 2 training regimens following comparing 2 training regimens following surgical repair of the knee for anterior surgical repair of the knee for anterior cruciate ligament damagecruciate ligament damage ““randomization method used was alternating randomization method used was alternating

assignment”assignment”

Referred to one patient as Referred to one patient as “difficult”“difficult” and and asked the friend to which group she was asked the friend to which group she was assignedassigned Friend responded Friend responded “that she was in the group “that she was in the group

that required slightly less contact and that her that required slightly less contact and that her inclusion was by his choice”inclusion was by his choice”

““Not so blind, after all”Not so blind, after all”

Sometimes subverting can work against a Sometimes subverting can work against a treatment’s apparent effectivenesstreatment’s apparent effectiveness

[Wallich P. Not so blind after all. Scientific American; May 1996.][Wallich P. Not so blind after all. Scientific American; May 1996.]

“Medical staff … may try to help their sickest patients by sneaking them into the treatment group instead of the control group.”

“Medical staff … may try to help their sickest patients by sneaking them into the treatment group instead of the control group.”

““The drug would then have to be significantly The drug would then have to be significantly better than conventional treatment just to appear better than conventional treatment just to appear equal in efficacy”equal in efficacy”

““The drug would then have to be significantly The drug would then have to be significantly better than conventional treatment just to appear better than conventional treatment just to appear equal in efficacy”equal in efficacy”

RCTs: Anathema to the RCTs: Anathema to the Human SpiritHuman Spirit

Scientific need to obtain unbiased data Scientific need to obtain unbiased data from an inherently biased source – from an inherently biased source – human beingshuman beings

Many trialists are unaware of the Many trialists are unaware of the rationale behind RCTs rationale behind RCTs

And the need for trial rigorAnd the need for trial rigor

Some may intellectually grasp the need, Some may intellectually grasp the need, but once immersed in a trial, have many but once immersed in a trial, have many contradictory interests (e.g. episiotomy contradictory interests (e.g. episiotomy trial in Canada)trial in Canada)

Anathema: A Psychologist’s ViewAnathema: A Psychologist’s View

””Fundamental conflict” between scientific Fundamental conflict” between scientific and human imperatives … “Many and human imperatives … “Many probably think they already know what probably think they already know what treatment is best.”treatment is best.” Geoffrey Scobie, psychologist at the Geoffrey Scobie, psychologist at the

University of Glasgow; Doctors cheat University of Glasgow; Doctors cheat because they know what’s bestbecause they know what’s best

New Scientist, 16 December 95, p. 10New Scientist, 16 December 95, p. 10

Of course, what we think we ‘know’ is Of course, what we think we ‘know’ is frequently wrongfrequently wrong

Reporting of Exclusions, Double-blinding, and Schedule Reporting of Exclusions, Double-blinding, and Schedule Generation Related to the Level of Allocation Concealment Generation Related to the Level of Allocation Concealment for 250 Trialsfor 250 Trials

Authors Authors

ReportedReported

Adequately Adequately ConcealedConcealed

(n=79)(n=79)

Unclearly Unclearly ConcealedConcealed

(n=150)(n=150)

Inadequately Inadequately ConcealedConcealed

(n=21)(n=21)

No ExclusionsNo Exclusions(Apparent)(Apparent)

53%53% 67%67% 76%76%

Double-blindingDouble-blinding 73%73% 39%39% 14%14%

Adequate generation Adequate generation of scheduleof schedule

29%29% 15%15% 0%0%