Page 1
Presented by Lee GiesbrechtOffice of Audits & EvaluationsInspector GeneralDepartment of Veterans Affairs
Presented by Lee GiesbrechtOffice of Audits & EvaluationsInspector GeneralDepartment of Veterans Affairs
StatisticalSamplingStatisticalSampling
2
What is Sampling?
• Check a small part of something to get an ideaabout the whole thing.
• For example…– Sip a spoon of soup to check the temperature of
the whole pot.
– Test a small sample of blood to check yourcholesterol levels.
– Other examples?
Page 2
3
Why Select a Sample?
Need to know something about a largepopulation, but its impractical to collect dataon every member of the population because:– The needed information about the population is
not on one central computer system or easilycompiled.
– It is too expensive or time-consuming to collectdata for all population members (travel, timeconstraints, personnel constraints, limitedbudget).
Two Kinds of Sampling
• Judgment or Convenience Sampling– Select sample units based on knowledge and
experience.– May result in representative data, but no way to prove
it scientifically.– Results cannot be generalized to the population.
• Statistical or Random Sampling– Introduce a component of random chance into the
selection process.– Can prove that data are representative (unbiased)
using statistical methods.
Page 3
Steps to Sampling
• Put audit goals into quantifiable terms –specifically what estimates of the populationare needed?
• Identify population of interest.
• Identify sampling frame(s) – what data areavailable for each population member?
• Design sample based on data needs andpractical concerns.
• Select sample – use random numbers.
6
Quantifiable Audit Goals
• Determine what estimates of the populationare needed – be as specific as possible.
• Do the analysis first – this will help to pinpointthe needed estimates.– Create “table shells” (empty tables) that will be
filled with numbers after data analysis is done.
– Identify subgroup analyses.
– Compare scenarios of possible outcomes tofurther help determine analytical needs.
Page 4
Statistical (Random) Sampling
• Sample selection is done using a random (notdeterministic) process.
• Each sample is one of a large number ofpossible samples.
• Results (i.e., means) from all possible samplestogether form a sampling distribution.
• The sampling distribution is centered on thetrue population mean.
8
Sampling Distribution
Page 5
9
What is Sampling Error?
• A.k.a. standard error, precision.• Difference between the projection and the
true (unknown) value in the population.• This difference is due to the randomness of
the sample.• Probability theory allows us to measure the
amount of sampling error for each projectionfrom our sample data.
• Use this to create a confidence interval (a.k.a.margin of error).
10
Computing Sampling Error
)1(
2
N
XXS i
Page 6
Sample Size
• Like most things, it is a balance of quality(precision), timeliness, and cost.
• Larger samples reduce sampling error.
• Decisions about the needed sample size take thesample design into account – stratification,clustering, and other complexities impact theprecision and, therefore, the needed size.
• In general, reducing sampling error by ½ requiresa four fold increase in sample size.
12
Sample Size and Sampling Error
ErrorRate
SampleSize
Plus orMinus
15% 50 8%
15% 200 4%
Page 7
13
Confidence Interval (Margin of Error)
Simple Random Sample
• Basis for all other sampling designs.• All members of the population have an equal
chance of being selected into the sample.• Easiest type of sample to analyze due to
simplicity of computing margins of error.• Often not practical to use in situations where
data are spread across geography.• Doesn’t take advantage of information you have
about the population to improveprecision/reduce the margins of error.
Page 8
15
A Little About “Randomness”
• Statistical sampling works when a sample isselected through a non-deterministic process.
• True random numbers cannot be generated bya computer – computers are completelydeterministic machines!
• True randomness is an “analog” process.
• Random numbers from a computer areactually pseudo-random – generated from aformula that can have a predictable outcome.
16
Simple Random Sample Example
• Use Random.Org to create a randomsequence of numbers.
• Copy random sequence column into samplingframe spreadsheet.
• Sort the list of records (sampling frame) by therandom sequence.
• Sample can be as large as needed, as long asrecords are used in order.
Page 9
17
18
Page 10
19
Random
Order
Original
OrderRO Number RO Name
7 1 301 Boston
10 2 304 Providence
9 3 306 New York
1 4 307 Buffalo
6 5 308 Hartford
5 6 309 Newark
3 7 310 Philadelphia
11 8 311 Pittsburgh
4 9 313 Baltimore
12 10 314 Roanoke
2 11 315 Huntington
8 12 316 Atlanta
20
Random
Order
Original
OrderRO Number RO Name
1 4 307 Buffalo
2 11 315 Huntington
3 7 310 Philadelphia
4 9 313 Baltimore
5 6 309 Newark
6 5 308 Hartford
7 1 301 Boston
8 12 316 Atlanta
9 3 306 New York
10 2 304 Providence
11 8 311 Pittsburgh
12 10 314 Roanoke
Page 11
Stratification
• Uses information from the sampling frame toforce the sample to be more representative ofthe population.
• Reduces sampling error because it ensuresthat the sample is more like the population.
• Form strata based on:– Amount/size of transaction
– Dates of transaction / type of transaction
– Location of sample units
Design Strata to…
• Ensure that the sample contains the sameproportion of population subgroups as exist inthe population.
• Ensure sample is spread across geography.
• Minimize the variance of a comparisonbetween groups.
Page 12
23
Stratification Example
• Stratified by percent incomplete C&P Exams.– Stratum A – highest percent
– Stratum B – middle percent
– Stratum C – lowest percent
• Ensures stations with different percentsincomplete C&P exams are in sample.
• Selected (almost) equal sized sample for eachstratum to minimize the variance ofcomparisons between strata.
24
Systematic Sampling
• Works when you have a complete list ofpopulation members.
• After determining sample size…
– Divide population by sample size (= interval).
– Select random number between 1 and interval.
– Select that record and every one after that byadding the interval.
• Can also sort frame for implicit stratification.
Page 13
25
Advantages/Disadvantages
• Simple Random Sampling– Advantage – Can quit work early – as long as sample
units are completed in order of selection.– Disadvantage – May be difficult to implement in the
field.
• Systematic Sampling– Advantages:
• Easier to select sample.• “Implicit” stratification by sorting list.
– Disadvantage – Can’t scale – must complete all sampleunits.
Clustered, Multi-Stage Samples
• Used when data must be collected at severallocations or there is no national frame.
• For example, we select a two-stage samplewith the first stage being a sample of officesand the second stage being claims folders.
• Sample is geographically “clustered” withineach sampled office.
Page 14
Multi-Stage Sample Example
• First stage sample of VISNs.
• Second stage sample of VAMCs.
• Third stage sample of appointments withineach sampled VAMC.
Effects of Clustering
• Reduces travel costs since fewer sites must bevisited.
• Reduces time required for data collection dueto less travel time.
• Generally reduces precision due to intra-cluster correlation.
• Two sources of sampling error:– Within cluster variation
– Between cluster variation
28
Page 15
Probability Proportional to Size
• Each unit sampled in proportion to its “size”.
• Examples of size measures:– Number of appointments
– Amount of payment
– Number of claims
• Ensures that “larger” units are more likely tobe sampled.
• Reduces sampling error for estimates relatedto the size measure.
Discovery Sampling
• Used to detect presence of rare events such asfraud and avoidance of internal controls.
• Select sample to have a specified probabilityof including at least one rare event based onan assumption of the prevalence of that eventin the population.
• If the rare item is found, can stop datacollection and report that the prevalence is atleast as high as that specified.
Page 16
Analysis of Sample Data
• Weighting
• Post stratification
• Compute estimates and sampling errors
• Interpret analytical findings
• Statistical significance testing
• Present analytical findings in audit report
What is a Sampling Weight?
• Sampling weights indicate how manypopulation members are represented by eachsample unit.
• In the simplest case, each sample unit has thesame weight and represents the same numberof population units (simple random sample).
• However, weights may not be the same for allsample units due to a more complex design.
Page 17
How are Weights Computed?
• Weights are the inverse of the probability of selection.
• For example:– Simple random sample of 100 units.
– Population of 5,000.
– Probability of selection is 100 in 5,000 = 1/50.
• Weight will equal the inverse, 50/1 or 50.
• Each sample unit represents 50 units in the population.
• Complex samples with unequal weights make thiscalculation more complicated.
34
Weighted vs. Unweighted Data
• With unweighted data, each sample unit iscounted equally.
• Unweighted data represent only those in thesample who provided data.
• With weighted data, each case is countedrelative to its representation in thepopulation.
• Weights allow analyses that represent thetarget population.
Page 18
Why are Weights Important?
• Weights allow us to calculate populationestimates.
• To estimate a population total, sum theweights.
• When weights for all sample units are thesame, population percentages equal samplepercentages.
• If weights are not the same for all sampleunits, must compute weighted percentages.
Why Unequal Weights?
• Stratification – i.e., each stratum has differentpopulation size but the same sample size.
• PPS sampling – each unit has a probability ofselection proportionate to its measure of size.
• Post-stratification – weights are adjusted aftersampling to conform to known populationtotals.
Page 19
Unequal Weights Example
StratumPopulation
SizeSample
SizeProbability of
SelectionSamplingWeight
1 5,000 100 1/50 50
2 8,000 100 1/80 80
3 2,000 100 1/20 20
4 12,000 100 1/120 120
Error Rate Example
Unweighted Error Rate: 79 ÷ 200 = 40%
Weighted Error Rate: 4,080 ÷ 14,000 = 29%
StratumPopulation
SizeSample
SizeSamplingWeight
ErrorsFound
WeightedErrors
1 2,000 100 20 54 1,080
2 12,000 100 120 25 3,000
Total 14,000 200 79 4,080
Page 20
Post Stratification
• Adjust the sample weights so that sampleestimates of known population values equalthose values.
• The population values used must be of higheraccuracy and precision than the sampleestimates.
• Improves accuracy and reduces bias of sampleestimates.
Prepare Data for Analysis
• Unless data were collected using a simplerandom sample, a micro data file (database)must be prepared.
• Basically, a spreadsheet where the rowscontain records and the columns containfields.
Page 21
What Estimates Are Needed?
• Error Rates– Overall – for the total sample.
– By strata or other characteristics.
– This is a frequency count of an attribute.
– Can get decent precision with a small sample.
• Impacts of Errors– Money that could have been saved.
– Time that could have been saved.
– This is a sum (mean, median) of a variable.
– Need larger sample to get decent precision.
41
Compute Sampling Errors
• If data are from a simple random sample,many programs will calculate this correctly.
• If data are from a complex sample design(stratified, clustered, unequal probabilities ofselection) must use other methods.– Taylor Series Linearization/Approximation –
approximates a linear function for a series of non-linear equations.
– Replication – uses the sample data to simulate theact of selecting multiple samples.
Page 22
43
Complex Sample Software
• SUDAAN – replication or Taylor SeriesLinearization.
• Stata – replication or Taylor Series Linearization.
• SAS – replication, ultimate clusters, directestimation (labor intensive programming).
• VPLX – replication (freeware created by Dr.Robert Fay at the Census Bureau).
• WesVar – replication.
• Others
What is Significance Testing?
• Demonstrate whether the difference betweentwo (or more) sample estimates is statisticallysignificant or if the difference could have beencaused by chance alone (sampling error).
• Takes the sampling error of the estimates intoaccount when comparing them.
• We sometimes compare estimates withanother number, for example, a numberreported by the audit client.
Page 23
How to Test for Significance?
• First compute the sampling error for eachestimate to be compared.
• Then compute the difference between theestimates.
• Then compute the sampling error of thedifference.
• If the confidence interval of the differencebetween the two estimates contains zero, thedifference is not statistically significant.
Significance Testing Example
Page 24
• All estimates mentioned in the report are listed in thesampling appendix with associated margins of error.
• The margins of error impact how estimates are stated.
• For example, if an estimate from the sample is1,624,981 transactions are in error with a margin oferror of 209,378, the report may state:– “About 1.6 million transactions are in error.”
– “About 1.6 million (± 200,000) transactions are in error.”
– “Between 1.4 million and 1.8 million transactions are inerror.”
Reporting Audit Results
Statistical Appendix
• Includes a short description of sampling andestimation procedures.
• Includes tables showing all estimates in thereport along with their margins of error.
• The confidence interval is stated in thesampling appendix – OIG policy is to use a 90percent confidence interval to compute themargins of error for sample estimatesreported in audits.
Page 25
Appendix Table Format
ErrorRates
EstimateMargin of Error Based
on 90% ConfidenceInterval
90% Confidence Interval
Lower Limit Upper Limit
Type 1 20% 8% 12% 28%
Type 2 25% 7% 18% 32%
Type 3 30% 6% 24% 36%
Note: the margins of error and confidence intervalsare indicators of the precision of the estimates. If alarge number of samples were selected and estimateswere computed from each one, 90 percent of thoseestimates would fall within the confidence interval.