Clinical laboratories are constantly changing tomeet customers' needs, operate on afinancially sound basis, successfully passproficiency programs and comply with themandates of regulatory agencies. Severalaspects of meeting customers' needs from alaboratory’s perspective are producinginformation which is accurate, precise andavailable in a timely manner. Laboratory
Med TechNet Tip
Click a topic to move quickly to the desired section.
Decision Making in the Clinical Laboratory Robert L. Klick
information which is both accurate and precise Elements of the law cover:allows members of health care delivery team tomake the best possible decisions concerningthe care and treatment of the patient.
Federal and state governmental bodies as wellas accreditation organizations now requireclinical laboratories to demonstrate anddocument the performance of the assays in useand to evaluate all non-exempt new assaysbeing introduced in the laboratory. The ClinicalLaboratory Improvement Act of 1988 (CLIA 88)published testing performance standards forindividual analytes. 1
The CLIA 88 analyte performance standardsare expressed as total fixed error limits or asstandard deviation total error limits. Implicit inthe published total fixed error limits is that themethod's combined inaccuracy and imprecisionshould be less than the allowable error limits.The intent of the analytical fixed error limits is toinsure that analytical errors will not invalidatethe medical usefulness of the test results andthat the method's performance comparefavorably the results form other laboratories.
The Joint Commission 1996 ComprehensiveAccreditation Manual for Pathology and ClinicalLaboratory Services has an objective ofImproving Organizational Performance (section1). In this section, improvements which benefit2
patients are listed. Essential activities toimprove the quality of patient care are:performance measurement, performanceassessment and performance improvement.This process of performance measurement,assessment and improvement can be applied tolaboratory methods evaluation and will bediscussed in this article.
CLIA 88 regulations regarding test performanceare covered in subparts J (Patient TestManagement) and K (Quality Control).
Test MethodologiesNormal Ranges (Reference Intervals)Test InterferencesPerformance Claims
AccuracyPrecisionSensitivity (Analytical Detection Limit)LinearitySpecificity (Interferences And Cross-reactivity)
If a new test is being considered forintroduction by the laboratory, a needsassessment should be done to determine theexpected number of analyses which will beperformed daily by the laboratory and when,during the day, specimens will be received andwhat the expected turnaround time is for thetest. The information gathered from the needassessment is essential in equipment andmethod selection as well as cost analysis.
After a needs assessment has defined theexpected workload and turn a round time, thelaboratory administration should establish aworking group to carry out the methodselection, evaluation, comparison, usertraining and implementation. The group shouldmake an objective search of vendors whosupply equipment and/or reagents and qualitycontrol materials for the new test. An excellentcomprehensive source of product informationcan be found in the Clinical LaboratoryReference published yearly as a supplementto (MLO) Medical Laboratory Observer,Medical Economics Publishing Inc., FiveParagon Drive, Montvale, NJ 07645-1742.Members of the working group should alsocontact and tap the collective wisdom ofcolleagues who are performing the test.Remember that these users have had toanswer many of the same questions with whichyou are now grappling.
The process of implementing a new testrequires a considerable amount of time,
Decision Making in the Clinical Laboratory Robert L. Klick
energy and cost, hence the administration must 4 (GraphPad Software) for calculations andbe prepared to make an investment in the graphs in this article.project. Depending on the regulations whichgovern clinical testing in a particular state and Within run precision is computed, by enteringthe test complexity, the laboratory will be the data in a column, selecting descriptiverequired to establish the performance statistics of the replicate determinations, thecharacteristics of the assay and may be mean (x) and sd along with other statisticrequired to successfully pass one or more information is calculated. The %CV is notrounds of proficiency testing prior to patient returned, but can be easily calculated bytesting. selecting a cell to place the result and typing
After acquiring the equipment, associatedreagents and supplies, the working groupshould become familiar with the use, calibrationof the assay and any special procedures (ex.dilutions) or data reduction required.
Precision Studies
After the familiarization phase, precision of themethod should be established. The least robustassessment of reproducibility is Within RunPrecision. Within run precision can be quicklyevaluated by running pooled sera/blood orquality control material multiple times toestablish reproducibility characteristics in termsof standard deviation (sd) and percentcoefficient of variation (%CV). A minimum oftwenty replications at medically importantconcentrations or activities should be assayedand used to calculate the above parameters ofprecision. The author highly recommends that apersonal computer with spreadsheet softwareand printer be available in each clinicallaboratory for data reduction, statistical analysisand graphic representation. Three excellentsoftware companies offering spreadsheetprograms are: Excel version 5 or later(Microsoft Corp.), Quattro Pro version 5 or later(Corel) and Lotus 123 (Lotus DevelopmentCorporation). All of these products areexceptionally good for use in the laboratory butthe buyer should investigate these and othersto insure that they contain the features the userdesires. I will be using Excel version 5 or Inplot
=(sd/x) x 100 and enter. (See Appendix 1,page 24, details.) Note, the numerical valuesfor sd and x can be entered into the equationor the cell locations from the descriptivestatistics output can be used.
Always inspect the results to determine if anyof the data points appears to be spurious (notdrawn from the same population as the otherobservations). If a value is markedly differentfrom the others, the possibility exists that thevalue is an outlier and can be eliminated fromthe replicate determinations. If the suspectobservation exceeds the mean ± 3sd, theprobability of this occurrence is 0.26% or onein 385 times by random change alone.Hence with a small sample size, the likelihoodof exceeding ± 3sd from the mean is small andthe questionable data point can be removedfrom the replicates. After the outlier is removedthe corrected mean, sd and %CV must becalculated.
What are some common causes of outliers inclinical analyses? Several possible causesinclude insufficient sample to allow the propervolume to be pipetted, a blockage of thepipette probe by specimen debris or acalculation or transposition error. Note, if in asmall number of replicates, two values exceedthe mean ± 3sd, the user should investigateand identify the cause of the disparate results,because it is highly unlikely that two suchobservations in a small population would occurby chance alone!
2 User sd 2 (n 1)Manufacturer sd 2
Calculated 2 (3.0 mg/dL)2 x 23(2.6 mg/dL)2
30.6
Decision Making in the Clinical Laboratory Robert L. Klick
Manufacturers often provide information about A clinical instrument manufacturer states thatthe performance of a new instrument or for total cholesterol the within run precisiontechnology in the form of within run, between (sd) was 2.6 mg/dL at 200 mg/dL. The userday or total precision. As users of the new determined the method’s within run precisionequipment, you should be assured that your to be 3.0 mg/dL in a replication study on aprecision is statistically no different than that of serum analyzer 24 times with a mean of 212the manufacturer. If the precision of your mg/dL. Is the user's precision the same as thatmethod is different (worse) than that of the of the manufacturer?manufacturer, a problem exists which needs tobe identified and corrected. The problem mayoriginate with the equipment, reagents, suppliesor the operators.
The statistical test which can be applied to test The critical Chi-square value at the 95%the user's precision versus that of the confidence level for 23 degrees of freedom ismanufacturer is the Sample VarianceCompared to Some Value, Chi-Squared Test.The Chi-squared calculated value ( ) is:2
Where n-1 degrees or freedom is the number ofdeterminations minus 1. The critical Chi-squarevalue for n-1 at a given confidence level(usually 95%) is obtained from a Chi -squaretable. If the calculated value exceeds the2
critical value, then the variance of the2
method as compared to the manufacturer'sclaim is different at that confidence level. Thereason for the difference should be exploredand corrected. If the calculated is smaller2
that the critical , than the variance of the2
method being evaluated is not statisticaldifferent than the manufacturer's claims. Theuser can conclude that precision of the methodhas been validated. Note the Chi-square testcan be applied to within run, between day andtotal precision estimates if comparableestimates are provided by the manufacturer.
EXAMPLE: Precision Data Verification
Example of verification of users precisionbased of manufacturer's precision data:
35.17, hence the calculated value (30.6) is2
less the critical value (35.17). The2
conclusion is that the precision of the methodis not different than the manufacturer'sprecision at the 95% confidence level.
Before evaluating the acceptability of thewithin run precision, based on the CLIA totalfixed error limits, you may wish to ask thequestion, is the precision of the new methodthe same as or different than the old method'sprecision. If the precision of the two methodsare different, it is advantageous for theprecision of the new method to be better thanthat of the old method and certainly not worse!The statistical test which answers this questionis the F-test for analysis of variance (sd ). The2
null hypothesis (question being evaluated)states that the variability (variance) betweenthe two methods are the same. The F-test canaccept the null hypothesis that the variances ofthe two method are the same at someconfidence level or reject the null hypothesis.Normally, the null hypothesis is tested at the95% confidence level, which implies that adifference in variances of the two methodswould occur by chance along 1 in 20 times orless (ex. 1 in 35 times).
F Larger (s.d.)2
Smaller (s.d.)2
4.022
2.9521.86
Decision Making in the Clinical Laboratory Robert L. Klick
Use the F-test to determine if a method’svariance is statistically equal to that of asecond method:
In the cholesterol evaluation study, the methodunder evaluation had a standard deviation of2.95 mg/dL and the comparison method (oldmethod) had a standard deviation of 4.02mg/dL (both estimates are based on replicationexperiments using the same control materialwhose mean cholesterol concentration was 199mg/dL). There were 10 measurements by themethod under evaluation and 17 measurementsby the old method.
Is the difference in precision between the twomethods statistically significant?
Calculate the F-value:
Degrees of freedom = n-1
Look up the F critical value 16 degrees offreedom for the numerator and 9 degrees offreedom for the denominator in the F table(Table 1).
F critical = 2.98 (p=0.05)
F calc. > F critical; Reject Null Hypothesis
F calc. < F critical; Accept Null Hypothesis
F calc. = 1.86 < F critical = 2.98; thereforethere is greater than a 5% probability ofobserving such a large difference in variancesby chance alone and although different, theyare not statistically different.
Med TechNet Tip
Use the "magnifying glass" tool to zoom in on tables and figures for a closer view.
Total Precision sd 2Within Run sd 2
Between Run
Decision Making in the Clinical Laboratory Robert L. Klick
Statistical estimates of imprecision (within run estimates includes the contributions of bothprecision) can be used as a first criteria of Within Run and Between Day Precision. acceptable or unacceptable performance.Within run sd should be less than the CLIAtotal error limit for the particular analyte beingevaluated. Ideally the random error(imprecision) should be 0.25 or 0.33 of the total Total assay precision should be used to decideerror limit. if the random error of the method meets the
Total Error Limits
Performance standards (total error limits) havebeen established using a number of criteria.One approach is based on the intra-individualvariation of an analyte. Fraser, has proposedthat analytical imprecision should be equal to orless than one half the normal intra-individualvariation. Additionally, CLIA 88 regulations3
define fixed limit goals in absolute terms ormultiples of standard deviations for a particularanalyte. Furthermore, as published by Koch,D.D. and Peters, P., a method’s total errorshould be less than one quarter the total errorlimit as defined under CLIA 88. Reasons for4
this criteria are, first, the chance of proficiencytesting failure approaches zero (assuming noinaccuracy in the method) and second, effectivequality assurance can be maintained with lessfrequent use of quality control testing materials.This latter consideration allows users to lowerquality control costs in the laboratory while stillidentifying significant changes in theimprecision of the method. If within runprecision is not acceptable, correct the problemor abandon the method.
It should be readily apparent that within runprecision estimates do not reflect an assay’simprecision in a realistic manner. A truerestimate of precision must take into accountassay imprecision as a function of time as wellas within run precision. Imprecision estimateswhich evaluate the effect of time can beachieved by determining Between DayPrecision. Furthermore Total Assay Precision
CLIA total fixed error limits. Note that the initialwithin run precision should first be used todecide if the method's reproducibility isacceptable given the CLIA allowable errorlimits or other error limits which users mayestablish.
The next section illustrates a concise statisticalapproach to the calculation of within andbetween as well as total precision estimatesfor a method. In this example a serum sampleis split and cholesterol was determined twicewithin a run (beginning and end of the run) oneach of a minimum of ten days. The simulateddata and associated calculations are shownbelow.
2 ( x1 x2 / 2)2
W
2
n87.025
102.95 mg/dL
Decision Making in the Clinical Laboratory Robert L. Klick
CLIA 88 CRITERIA FOR ACCEPTABLE PERFORMANCE(TOTAL FIXED ERROR LIMITS)
ANALYTE ACCEPTABLE PERFORMANCE
Erythrocyte Count Target Value ± 8%
Prothrombin Time Target Value ± 15%
Cholesterol Target Value ± 10%
Blood Alcohol Target Value ± 25%
-Fetoprotein (Tumor Marker) Target Value ± 3 SD
Sodium Target Value ± 4 mmol/L
Calcium, Total Target Value ± 1 mg/dL
IgE Target Value ± 3 SD
IgG Target Value ± 25%
Total precision of the method is estimated by combining both within run and between day precision.Total standard deviation ( ) is calculated as the square root of within run variance plus between runT
variance.5
The following Table contains examples of thecriteria for acceptable performance (total fixederror limits) as published in the CLIA 88Federal Register, Vol. 57, No. 40, February28:7149-68,1992. The complete list ofperformance criteria can be found in thereference cited above. Note the performancecriteria are expressed in several differentways; first, as an absolute value in the units ofthe analyte, second, as the target value ±multiples of the standard deviation for thatanalyte or the target value ± a percent (%) ofthe actual value. Each manner of expressingacceptable criteria is illustrated in the tablebelow.
Is the cholesterol method acceptable in termsof Random Error? Listed above is the totalerror limit for serum cholesterol. The allowableerror is the target value ± 10%. Using theexample above, the serum cholesterol targetvalue was 200 mg/dL. The total error is 200mg/dL x 0.1 = 20 mg/dL and the acceptabletotal error limits is 180 to 220 mg/dL.
First, always determine if within run isacceptable. If the precision is not acceptable,the random error must be reduced or themethod should be abandoned. If the within runprecision is acceptable, determine if totalprecision is acceptable. The goal is to have
Inaccuracy as % Deviation
x measured Target ValueTarget Value
X 100
Decision Making in the Clinical Laboratory Robert L. Klick
imprecision less than the total error limit or amuch better criteria is or ¼ total error limitgreater than the method's sd:
In the cholesterol data above the within runprecision was 2.95 mg/dL at 200 mg/dL. Threetimes within run precision = 8.85 mg/dL (3 x2.95 mg/dL), which is much less than the CLIAtotal error limit of 20 mg/dL at 200 mg/dL.Within run precision is acceptable. The totalprecision estimate was 3.58 mg/dL. Threetimes the total precision = 10.74 mg/dL. Thisvalue is again much less than the total fixederror limit of 20 mg/dL. The precision of thismethod is satisfactory to meet the CLIA goalsand to assure the user that the probability offailing proficiency testing is very small,assuming no inaccuracy. 6
Meeting CLIA Criteria
With assurance that the method's precision isacceptable, next the method must beevaluated to determine if accuracy meets CLIAcriteria. Accuracy verification can bedetermined using calibration materialstraceable to the National Institute of Standardsand Technologies (NIST), assayed controlmaterial and proficiency testing materials(http://www.nist.gov/). All must be compatiblewith the method under investigation. 7
A simple protocol for estimation of inaccuracyinvolves analyzing one of the materialsreferenced above, a minimum of three times tominimize random error and calculating themean. It is essential that the concentration oractivity of the material analyzed be within the
linear range of the assay! A point estimate ofthe method's inaccuracy is the target valueminus the mean analyzed value. Thisdifference divided by the target valueexpressed as a percentage is the % deviation.
Ehrmeyer and Laessig have publishedAccuracy Verification Tolerance Limits formany commonly performed analytes. The total7
cholesterol accuracy verification tolerance limitis ± 4%. In the example above, the targetconcentration of the cholesterol control was200 mg/dL. The method under evaluation hada mean for all days of 203.3 mg/dL. Thedeviation from the expected value of 200mg/dL was 1.65%. The inaccuracy (1.65%) at200 mg/dL is less than the ±4% accuracytolerance limit, hence the conclusion is that theassay meets accuracy goals. Note, accuracy ofan assay should be assessed at all importantmedical decision concentrations. If theinaccuracy of the assay exceeds the accuracyverification tolerance limit, the problem mustbe corrected. One of the first corrective actionsto be taken is to recalibrate the instrument. Then, new standards should be employed incalibration.
Reportable Range
The Reportable Range of an assay isdetermined by linearity studies. Implicit in thereportable range is determining the LeastDetectable Dose (LDD) and Maximum DoseLimit (MDL). Determination of the reportablerange can be done using commerciallyavailable linearity materials with analyte valuestraceable to NIST standards, assayed qualitycontrol materials or clinical samples with bothlow and very high analyte concentrations
Decision Making in the Clinical Laboratory Robert L. Klick
which have been accurately diluted and was determined by regressing the five lowestanalyzed multiple times to obtain mean values. target concentrations versus the mean
The following example using cholesterol concentrations to determine the regression linelinearity materials illustrates the calculations of allows for a visual inspection to determine theMDL. The assigned target cholesterol distance a value is from the line.concentration of the linearity solution was 800mg/dL. Summarized in the table is the dilution Least detectable dose (LDD) is in generalmethod, calculated target value, measured defined as the measured response at zeroconcentrations, mean concentration and dose of the analyte ±2 sd. If the calibrationpercent deviation from the target value. curve has a positive slope, then LDD is the
Using the maximum allowable inaccuracy of±4% of the target value, it is apparent that thedeviation is unacceptable (-8.3%) at a totalcholesterol of 800 mg/dL. Hence the upperlimit of linearity (MDL) is 640 mg/dL. The CLIAtolerances can be used to determine at whatanalyte concentration the deviation exceedsthe total error limit, but please realize that thisapproach does not allow for random error!Investigators have suggested that in generalthe allowable deviation should not exceed 5%of the target value. In the example givenabove, the MDL would be the same using the±4% or ±5%.
A graph of the relationship between the targetvalue and the measured values should alwaysbe plotted for visual inspection to confirm the 4 sd. The following table summarizes thecalculated deviations. Figure 1 is a plot of thelinearity data. The regression line of this plot
measures values. Using the lower
7
response at zero +2 sd and if the calibrationcurve has a negative slope, then LDD is theresponse at zero -2 sd. The concept of theminimum ±2 sd is acceptable for generalclinical testing, but if the information is orcould potentially be used in a medicolegalcontext (ex. urine drug screening) then thelaboratory must insure that the probability ofthe false positive in extremely low. If thelaboratory must guard against a falsepositive result, a more appropriate estimateof LDD might be the response at zero +3.5 or
probability that a truly negative (zeroconcentration) specimen would exceed a
Decision Making in the Clinical Laboratory Robert L. Klick
Probability of False Positive asMultiples of SD above Zero
> Zero + 2 sd 2.28% 1 in 44 times
> Zero + 3 sd 0.13% 1 in 769 times
> Zero + 3.5 sd 0.02% 1 in 5000 times
> Zero + 4 sd 0.003% 1 in 33,330 times
stated sd above the mean by random chance cholesterol aqueous calibrators which arealone (false positive). viscosity adjusted to simulate sera may be the
The probability of a truly negative specimen zero, then LDD = Response at Zero Dose + (2being a false positive due to chance alone is x sd).2.28% using the detection limit of zero +2 sd.Stating the concept another way, if a negativespecimen is analyzed repeatedly, one in 44times a result would equal 2 or more sd abovethe response at zero dose. For routine clinicalanalytes, the response at zero dose +2 sd isan acceptable criteria for LDD.
For elicit drug screening in the workplace orother medicolegal applications a 1 in 44 falsepositive results would be disastrous. A moreappropriate criteria of LDD might be zero +3.5sd. Using the zero plus 3.5 sd criteria of LDD,only 1 in 5,000 truly negative specimen wouldequal or exceed the limit and be a falsepositive result. Laboratories can set LDD bychoosing the most appropriate multiple of thesd for their situation.
LDD is determined by analyzing a control orspecimen with a very low analyteconcentration or activity multiple times (20times is ideal but no less than 10 times) andcalculating the sd. The specimen selected forthe replicate testing should have a matrix assimilar to the actual samples as possible. Forexample, it would be inappropriate to analyzea serum based specimen, if the actual clinicalsamples were urine. Unfortunately it is manytimes difficult or impossible to find specimenswith very low analyte concentrations, hencecompromises are made. For example, total
best available possibility. A delipidated serumpool would be better but more difficult toobtain.
As an example, a 25 mg/dL cholesterolcalibrator was analyzed 20 times. The resultsare: 23, 22, 25, 28, 21, 24, 24, 25, 27, 28, 24,23, 25, 25, 20, 28, 22, 27, 25 and 29. The sd ofthe replicates is 2.5 mg/dL and is an estimateof imprecision as a very low concentration. Ifthe assayed concentration is zero at a dose of
LDD = 0 mg/dL + (2 x 2.5 mg/dL) = 5.0 mg/dL
The analytical range of the cholesterolmethod extends from 5 to 680 mg/dL. Thelinearity has been assessed at low, normal andhigh physiological concentrations to establishif accuracy is acceptable at these levels. Inmost cases the analytical range for a methodmay be used as the Reportable Range. Onecautionary note is that if the matrix (eg,viscosity and/or ionic strength) of the linearitymaterials or the material used for LDD aredifferent than the clinical specimen, thereportable range and analytical range may besomewhat different. Many instrumentmanufacturers publish or incorporate in theequipment algorithms for reportable ranges.The manufacturers’ reportable ranges shouldstill be confirmed using a evaluation protocolsimilar to the one outlined in this presentation.It is always good laboratory practice toestablish the performance of the method inyour laboratory and never accept withoutconformation the claims of manufacturers orother laboratories.
Recovery Studies
Recovery Studies are used to establish whatportion (percent) of the analyte in thespecimen is being measured. Ideally, the exact
Spiked Aliquot 1.9 mL QC sera 0.1 mL cholesterol, 800mg /dL
Baseline Aliquot 1.9 mL QC sera 0.1 mL of diluent (saline )
Concentration Analyte Added High Analyte Conc. X vol. Analytetotal volume
amount of analyte in the specimen should bedetected by the method and recovery would is100%.
If the recovery is different from 100%,proportional error exists. Proportional error atimportant medical decision concentrations maycompromise the usefulness of the method.
To conduct a recovery study, an appropriatespecimen is divided in two. To a known volumeof the aliquot, a known volume of a highanalyte concentration or activity is added(spiked specimen). Next, to a equal volume ofthe aliquot, a known volume of an appropriatediluent is added (baseline specimen). It isimportant to keep the ratio of diluent tospecimen volumes as small as possible.Ideally, the volume of analyte or diluent shouldbe no greater than 10% of total volume tominimize dilutional (matrix) effects relative toneat clinical specimen.
EXAMPLE: Recovery Experiment
An easy way to prepare the spiked aliquot is touse the highest linearity material available or apatient specimen with a high concentration.For this example, the high linearity materialhad a cholesterol concentration of 800 mg/dL.The specimen aliquot was a quality control(QC) material with a normal cholesterol level. Preparation of Baseline and Spiked Aliquot
Next the two aliquots are analyzed in duplicateor, better still, in triplicate and the mean valuescalculated.
The concentration recovered is the spikedaliquot concentration minus the baselinealiquot concentration.
The concentration of the analyte added iscalculated as the concentration of that added,times the volume added divided by the totalvolume of the aliquot.
The mean recovery for the method iscalculated as the concentration recoveredtimes 100 divided by the concentration added.
In the example above the recovery was 97.5%.Is that magnitude of proportional erroracceptable? To determine acceptability of theproportional error (PE), a medical decisionconcentration must be stated. Let us evaluateproportional error at 200 mg/dL, the cutoffbetween normal risk and moderately elevatedrisk of development of cardiovascularcomplications. Using the CLIA total error limitof ± 10% of the target value, at 200 mg/dL, ±20 mg/dL in acceptable. The point estimate ofproportional error is mean recovery (as afraction) times the medical decisionconcentration subtracted from the medicaldecision concentration. 8
Decision Making in the Clinical Laboratory Robert L. Klick
A proportional error of 5 mg/dL is clearly much Provided below are protocols for theless than the CLIA total error limit of 20 mg/dL preparation of the interferants: lipids,at 200 mg/dL, hence the conclusion is that hemoglobin and bilirubin.recovery is acceptable.
Interference Studies
Interference Studies are performed to assessthe effect of compounds which potentially altertest results in either a negative or positivemanner. Commonly evaluated interferingcompounds include: hemoglobin (hemolysis),lipids (lipemia), bilirubin (icterus),anticoagulants and preservatives, vitamins(dietary supplementation), analgesics (aspirinand acetaminophen), drugs (ex. lithium) anddrug metabolites. Interference studies arecarried out in much the same manner as arerecovery studies. In interference studies aknown amount of the interfering compound isadded to a specimen to produce a samplewhich contains a high concentration of theinterferant. In general, drugs should be testedat 5 to 10 times the upper limit of thetherapeutic range. Preservatives andanticoagulants should be evaluated at twicethe normal additive specimen concentration,thus simulating a "short draw" in phlebotomy.An excellent source of information aboutinterfering compounds and recommendedevaluation procedures can be found theNCCLS EP7-P guidelines. 9
In immunoassays, in addition to the potentialinterferences listed above, users need toprovide information about the specificity(cross-reactivity) of the antibody with closelyrelated antigens or haptens. Mostmanufacturers of immunoassays provide fairlyextensive assay specificity information, butbecause of a unique patient population,laboratories may need to evaluate potentialcross-reactivity of a new drug.
Lipid Solution, 1,000 mg/dLA simulated lipemic serum specimen can beeasily produced by obtaining an I.V. fatemulsion solution. One such preparation isIntralipid, 20% (w/v), KabiVitrum Inc, Clayton,NC 27520. This solution contains 20 g ofsoybean oil per 100 mL. The addition of 0.5mL of Intralipid and 9.5 mL of water (reagenttype I) to a 10 mL lyophilized normal controlwill produce a serum based material with anadded triglyceride concentration of 1,000mg/dL. This preparation will simulate a fairlylipemic serum specimen, but a greater amountof Intralipid can be added to produce a higheradded triglyceride concentration.
Hemoglobin Solution, 5000 mg/LHemolysis in a serum or plasma specimen canbe prepared by the addition of freehemoglobin. To prepare a solution of freehemoglobin collect a tube of heparinizedblood. Centrifuge, 1000 xg for 10 minutes, theblood as you normally would to obtain plasma.Decant the plasma and add isotonic (sodiumchloride, 0.156 mol/L) saline at a volumeapproximately equal to twice the packed RBCvolume. Gently re-suspend the RBCs,centrifuge, 1000 xg for 10 minutes, again anddecant the supernatant. Repeat the salinewash three more times as indicated above.After the final centrifugation and decanting ofthe supernatant, add reagent type I water of avolume equal to the volume of the RBCs andmix. The RBCs can be lysed by mechanicaldisruption (ex. tissue grinder) or refrigerationovernight (hyposmotic manipulation). Thesolution is next centrifuged at 2000 xg for 30minutes to remove the stroma.
Analysis of the free hemoglobin can be doneusing the Drabkin's cyanomethemoglobinmethod or a direct spectral method. Several
Free Hemoglobin, g /L {0.836[ 2Abs415 (Abs380 Abs450 ) ]} x 100
Vol Hb Soln Desired Hb. (mg /L)Hemolysate(mg /L)
X Tot. Vol.
Vol Hb Soln (mL) 5,000 mg/L70,000 mg/L
X 10 mL
0.714 mL
Decision Making in the Clinical Laboratory Robert L. Klick
direct spectral methods for the determinationof free hemoglobin concentration arediscussed by Fairbanks, Virgil F. et al. One10
method described in the publication requiresthat the plasma be diluted 11-fold with asodium carbonate solution, 0.942 mol/L, priorto determining the absorbance at 380 nm, 415nm and 450 nm against the sodium carbonatesolution. Since the hemolysate is tooconcentrated a further 100-fold dilution isrequired.
To make the appropriate dilution to yield a finalhemoglobin concentration of 5000 mg/L, applythe following equation.
EXAMPLE: Interferant Preparation
If 10 mL of a 5,000 mg/L hemoglobin solutionis required and the hemolysate hemoglobin is70,000 mg/L, what volume of the hemolysatemust be diluted to a final volume of 10 mL withreagent grade water?
Ten mL of a 5,000 mg/L hemoglobin solution isprepared by diluting 0.714 mL of hemolysate tothe final volume. Ten mL of the 5,000 mg/Lhemoglobin can be added to reconstitute alyophilized 10 mL normal control.
Bilirubin Solution, 20 mg/dLCommercially available bilirubin solutions ofapproximately 20 mg/dL can be purchased andused to reconstitute normal control materials.This approach is simple and easy, butexpensive since relatively large volumes arerequired to reconstitute a control vial. Asecond approach is to prepare a 20 mg/dLbilirubin solution from crystalline bilirubin.Highly purified crystalline bilirubin can bepurchased inexpensively from many chemicalvendors.
Weight out 20 mg of crystalline bilirubin on aplastic weighing boat and transfer to a 100 mLvolumetric flash. Dissolve the bilirubin byadding 1.0 mL of dimethylsulfoxide (DMSO).Note, DMSO should also be used to wash theweighing boat of any residual bilirubin.Completely dissolve the bilirubin by swirlingand next add 2.0 mL of sodium carbonate, 0.1mol/L and add approximately 70 mL of reagentgrade water. Adjust the solution to a final pH of7.4 by drop-wise addition of 0.1 mol/Lhydrochloric acid while monitoring with a pHelectrode and meter. Finally dilute to 100 mLwith reagent water. This solution has abilirubin concentration of 20 mg/dL. 11
Using a lyophilized 10 mL normal control,reconstitute the control vial by the addition of10 mL of the bilirubin solution, 20 mg/dL. Thiscontrol material should be stored in the dark toprevent light mediated degradation.
Finally, a normal control of the same lot asused to prepare the interference solutionsshould be reconstituted with 10 mL of reagentwater. The mean values of this controlanalyzed in triplicate will serve as the analytetarget concentrations or activities. The normalcontrol materials containing the added lipid,hemoglobin and bilirubin should also beanalyzed in triplicate and the mean valuesdetermined. A point estimate of the effect of
each interferant on the assay can be determines cross-reactivity but is moredetermined. complicated to carry-out. The following
Using this equation to calculatepercent interference, both positive andnegative types of interference in the assay canbe quantified at known levels of the interferingsubstance.
EXAMPLE: Calculation of Interference
The mean total cholesterol in the target controlwas 199 mg/dL and the mean total cholesterolin the control with 5,000 mg/L hemoglobinadded was 206 mg/dL.
In this case, hemoglobin at 5,000 mg/L falselyincreased the target total cholesterolconcentration by 7 mg/dL or 3.5%. This level ofinterference is acceptable given the CLIAguidelines of ± 10% but just acceptable at atotal fix error of 4%. Calculation of the percentinterference due to the other interferingsubstances are determined in the samemanner.
Evaluation of immunoassay cross-reactivitycan be evaluated in several different ways.One method, employed in competitiveimmunoassays, replaces calibrators with thepotential interfering compound as assaycalibrators. The calibration curves areconstructed and analyzed to determine theconcentrations at which a 50% displacement ofthe labelled antigen or hapten from theantibody occurs. This method accurately
protocol is simple and in most cases, providesadequate estimates of antibody cross-reactivity.
Evaluation of Immunoassay Cross-reactivity
Obtain:
A. A sample negative for the analyte (ie.drug or hormone)
B. Potential interferant of knownconcentration
Prepare Solutions:
A. 0.1 mL Diluent + 0.9 mL Sample (zeroconc.), mix
B. 0.1 mL interferant + 0.9 mL of Sample(zero conc.), mix
Analyze both solutions in triplicate, calculate x
Example:
Comparison Studies: Final Phase
The final phase of a methods comparisonstudy is to analyze actual clinical samples
Syx
(yi Yi)2
n 2
Decision Making in the Clinical Laboratory Robert L. Klick
which have the broadest possible analyteconcentration range. The team should becollecting these specimens in advance of thefinal phase of the evaluation. Ideally 120specimens should be analyzed, but manyevaluations are done with fewer samples. Arandom sampling of 128 abstracts publishedfor the 1997 annual meeting of ClinicalChemistry revealed that the mean number ofclinical specimen analyzed in methodscomparison studies was 101.8 and the medianwas 63. Larger numbers of samples in the12
comparison study provide more confidence inthe parameters which describe the relationshipbetween the methods.
Unique specimens which contain potentialinterferences should also be included in thestudy. It is imperative that the range ofconcentrations or activities in the sample bebroad. Several investigators suggest thatadequacy of range can be evaluated with thecorrelation coefficient (r). Cary, R.N. et at,have suggested that r > 0.99. Note, that the8
correlation coefficient (r) is used toestablish the adequacy of analyte rangeand not to reach conclusions about theperformance of the methods. Someinvestigators have used r as a primarydescriptive parameter in evaluatingrelationships between methods. A better andmore quantitative parameter to evaluaterandom error between methods is the standarddeviation of the residuals about the regressionline which is also called the standard error ofthe estimate (S ). The calculation and use ofyx
S are discussed in a following section.yx
Three types of error can be observed betweenmethods. They are random error, constanterror and proportional error. One or more ofthese errors may be observed betweenmethods. By evaluation of the types andmagnitude of the errors, decisions can bemade about the acceptability of clinicalmethods.
Random Error
Random error is due to imprecision ofmeasurement of the methods and is randomlydistributed about the regression line. Themagnitude of random error can bequantitatively estimated by the standarddeviation of the residuals, (S ). S isyx yx
calculated as the square root of the sum ofsquares of y at the line minus the actual yvalue divided by n-2.
where: y = actual y valuei
Y = y at the linei
Most all statistical software packages willcalculate S , but some spreadsheet programsyx
will not directly calculate S under theyx
regression option. Hence an estimate of S yx
can be obtained by requesting residuals,standard residuals (residuals in sd multiples)and a residual plot. From the residuals, thestandard deviation of the residuals can beobtained by using the descriptive statisticfunction. A good estimate of S can beyx
obtained in this manner when the number ofsamples exceeds 30. In the random error datapresented in Figure 2 with 38 samples in thecomparison, the correct calculation of S wasyx
14.4 and the spreadsheet calculation of S yx
was 14.5, an error of less than 1%. The reasonfor the error is that the spreadsheet uses n-1 inthe sd calculation rather than n-2 as requiredin the correct S calculation. For comparisonsyx
using smaller numbers of sample, the error inestimation of S using n-1 in the calculationyx
becomes larger and the equation above shouldbe used.
Excel 5 will calculate S by using the STEYXyx
command in function wizard. The format would
Decision Making in the Clinical Laboratory Robert L. Klick
be STEYX({y-values},{x-values}) and willreturn S . An example,is:yx
STEYX({5,7,9,11,13,15,17,12},{6,8,11,12,13,18,15,14}) = sSTEYX({5,7,9,11,13,15,17,12},{6,8,11,12,13,18,15,14}) = s = 1.629. = 1.629. yxyx
Also Excel 5 calculates the standard error ofthe regression line (S ) when regressionyx
analysis is performed.
When graphing methods comparison data, it iscustomary to plot the reference or existingmethod on the x-axis and to plot the methodunder evaluation on the y-axis. Also it isconvenient to graph both axes using the samescale, in this manner the plots are square anda line from the lower left to top right cornerrepresents perfect agreement betweenmethods. These conventions make is easy toevaluate the types and magnitude of errorwhich may occur.
Figure 2, illustrates appreciable random errorbetween the two methods. Note the scatter ofthe points about the line, S equals 14.4. Unitsyx
of S are the units of analyte concentration oryx
activity. Figure 3, illustrates a reduction inrandom error, S equals 7.79. The S ofyx yx
Figure 3 is approximately half that observed inFigure 2. Visual comparison of the two figuresreveals that Figure 3 has data points clusteredmore closely about the line. As random errordecreases S decreases. The slope in bothyx
Figures 2 and 3 is close to 1.000 and the y-intercept is close to zero.
Figure 4, illustrates the effect of concentrationrange on the correlation coefficient. InFigure 2, r = 0.9836, the range of x-valueswas 48 to 333 with an S of 14.4. An r of lessyx
than 0.99 indicates the range is not broadenough to insure adequate estimates ofrelationships between methods. In Figure 4, ris now greater than 0.99 (r= 0.993) with asimilar S of 15.0. The main differenceyx
between Figure 2 and 4 is that the range ofsample analyte concentrations has beenextended to 489 from the previous maximum of333. Extending the range increased thecorrelation coefficient (r) when the randomerror (S ) remained essentially the same in theyx
two comparisons.
Decision Making in the Clinical Laboratory Robert L. Klick
An important point to remember about leastsquares regression analysis is that the modelassumes that the x-axis values (independentvariables) are the true concentration andassigns all error of estimation to the y-axisvalues (dependent variables). In reality both xand y methods have some degree ofimprecision and a better regression modelwould partition the imprecision ofmeasurement between both the x and yvariables. The Deming's regression modelrecognizes this problem of imprecision inanalytical methods and gives a betterregressional relationship when both methodshave appreciable random error. 13
Recently observed comparison data from twoserum total iron binding capacity methodsrevealed a slope of approximately 0.91 and asizable bias by least squares linear regression.Deming's regression yielded a slope ofapproximately 0.99 and insignificant bias. Totaliron binding capacity is one of the least precisetests done in clinical chemistry and randomerror should be attributed to both methods.Deming's regression was the more appropriate
model is this case. If the imprecision of the xmethod is small then only minor differenceswill occur between the two regression models.
Most commercial spreadsheets only performleast squares regression, so if Deming'sregression is required, special software willneed to be purchased or the data sent to acompany for data analysis. I would recommendthat both least squares and Deming'sregression software be available for use in thelaboratory.
Identification of outlier data points about theregression line should begin with the a visualinspection of plotted data. Figure 5, reveals byvisual inspection one observation (indicated bythe arrow) is suspiciously different from allother points about the line.
Is this observation spurious for some reasonand what criteria should be applied to acceptof reject the data point as part of thepopulation? A good criteria for rejecting the
Decision Making in the Clinical Laboratory Robert L. Klick
data is when the point is greater than 3.5 timeS form the regression line. The data point inyx
question is 175 by the reference methodequals and 260 by the evaluation method (175,260). Solving for what the evaluation methodshould have given (value at the regressionline) when the reference method was 175,yielded 174.86.
y = mX + b = (0.991)(175) + 1.43 = 174.86c
where: m = slopeX = x valuec
b = y-intercept
The actual value (260) minus the value at theline (174.86) equals 85.14. This y-axisdistance from the line (85.14) divided by S yx
(20.16) equals 4.22 times S from the line. Theyx
data point in question is 4.22 times S from theyx
regression line and clearly exceeds 3.5 S .yx
The point can be rejected as an outlier.
Figure 5 contained the same data as in Figure2, except for the inclusion of the outlier. Thisone spurious point increases S by 5.76 unitsyx
(40%) from 14.4 to 20.16. After an outlier hasbeen rejected it is essential that theregression analysis be repeated to establishthe corrected equation of the line andassociated parameters.
Proportional Error
Proportional error produces results by onemethod which are some multiple (percentage)of results by the second method. Proportionalerror is evaluated by assessing the slope ofthe regression line. If the slope at someconfidence interval is different from 1.00,then significant proportional error existsbetween methods. In proportional error, S yx
increases with the magnitude of the error butthe y-intercept will not be significantly differentfrom zero.
Figure 6, illustrates proportional error. Theslope is 1.188, which implies that the averagevalue for the evaluation method (y-axis) was1.188 times that of the reference method (x-axis). The 95% confidence interval of the slopeis 1.11 to 1.26 and does not include 1.00,hence we conclude that proportional errordoes exist. S is very large at 17.65 alsoyx
indicating proportional error. The y-intercept isnot significantly different from zero (y = -0.78).The reasons for proportional error are first thatone method is not completely measuring allthe specimen analyte. Recovery studiesdiscussed earlier in this paper will help confirmthis possibility. Recovery in one method willbe significantly less than 100%. A secondpossible cause of proportional error iscalibration error of one or both methods.Recalibration of one or both methods mayrectify the problem of proportional error.
The bias (x - y) is the difference betweenmethods at the means. The bias should beless than the CLIA total error limit. In thisexample, bias equals 203 - 172 = 31. The
Decision Making in the Clinical Laboratory Robert L. Klick
CLIA total error limit is 10% of the target value.The mean x was 172 and 10% of this value is17.2. Clearly proportional error at the mean isunacceptable (31 is greater than the allowableerror of 17.2).
Systematic error may be described as theinaccuracy of a method when compared toa method with established accuracy. Bias isone measure of systematic error. In additionto the inaccuracy at the mean, it is important tocalculate the systematic error at medicaldecision concentrations. For total cholesterol,laboratorians would like to be able to assureusers that systematic error is acceptable at200 mg/dL, the borderline between normal andmoderately increased cardiovascular risk.Systematic error (SE) is the difference atthe medical decision concentrationbetween the reference method and theevaluation method obtained by regressionanalysis. 8
Systematic Error (SE) = (mX + b) - Xc c
Where:m = slopeX = Medical decision concentrationc
b = y-intercept
Using Figure 6 at an example, at a totalcholesterol of 200 mg/dL, the systematic erroris 36.9 mg/dL.
The CLIA total error limit at 200 mg/dL is 20mg/dL. Clearly the systematic error of 36.9mg/dL is greater than the recommended totalerror limit (20 mg/dL) and the performance ofthe method is unacceptable.
Constant Error
A third type of analytical error which can existbetween methods is Constant Error.
Constant error is caused by interferences inthe analytical samples. These interferenceswill produce a constant difference regardlessof analyte concentration between theevaluation and reference methods. The natureof the interference may be either positive ofnegative with respect to the reference value.Using regression analysis, Constant error ismanifest by a y-intercept which issignificantly different from zero.
Figure 7, illustrates constant error between thereference and evaluation method. The y-intercept is 19.3 and the 95% confidenceinterval of the intercept is 7.8 to 30.9. Theconfidence interval does not include zero. The95% confidence intervals of the slope and y-intercept are shown in Figure 7 as the dashedlines. Note the 95% confidence intervals do
Decision Making in the Clinical Laboratory Robert L. Klick
not include zero at the y-intercept. Evaluators need to be established for a definedare fairly assured that this magnitude of off-set population. Judgement of the medicalin the y-intercept would not happen by chance significance of mean differences would need toalone. The cause of the constant error may not be considered. The statistical test to bereadily be apparent by regression analysis, butinterference studies discussed earlier in thepresentation will be helpful in the elucidation.Systematic error is attributable constant errorin Figure 7, at a medical decisionconcentration of 200 mg/dL total cholesterol is17.3 mg/dL. Total allowable error is 20 mg/dLat 200 mg/dL, so the systematic error isacceptable, but inaccuracy consumes nearlyall of allowable error!
The final criteria of acceptable methodperformance is total error. Total error is ameasure of systematic plus random errorand should be less than the CLIA total errorlimit.
Total Analytic Error = Systematic Error + 3 x sd total
Total Analytic Error = Systematic Error + 4 x sd total
The total analytic error criteria of SE plus 4 xsd is the ideal measure of a method'stotal
performance. If total analytic error, calculatedby this equation is less than CLIA total errorlimit, then the chance of proficiency failure isremote. Total analytic error calculated as SEplus 3 x sd is still a robust measure oftotal
performance. 8
Final Performance goal!!
Total Analytic Error < CLIA Total Error Limit
A final question needs to be asked: are thepatient sample means by the two methods thesame? If the means are the same statistically,then the existing reference intervals can besafely used for the new method. If the meansare different, then new reference intervals may
employed is the Paired-t test of samplemeans. Using the data from Figure 2, wherethe analytical error was primarily random, thetwo-tail probability (p=0.322) is 32 times out of100 the differences in means would occur bychance alone. The Excel printout of the paired-t is shown in Table 2.
A generally accepted significant difference inmeans occurs when p <0.05. Hence theconclusion is that there is no significantdifference in the means of the two methodsand that the same reference intervals could beapplied for the new method.
Table 3 contains the results of the paired-t testfor the data used in Figure 6. The methods
Decision Making in the Clinical Laboratory Robert L. Klick
comparison revealed proportional error with a assures laboratorians that results will satisfyslope of 1.1884. The two-tailed probability medical care needs and pass proficiency(p=1.62 ) is much less than p=0.05, hence testing in a consistent manner. -10
the null hypothesis is rejected. The two means Finally, performance improvement should beare not the same and a new reference interval an important part of method evaluation andwould most probably need to be established. selection by the laboratory. Precision and
Summary
Underlying this paper, has been the goal, asstated in the Joint Commission ComprehensiveManual of Performance Measurement,performance assessment and performanceimprovement. Covered in this paper are many2
objective techniques to measure performancein terms of precision, accuracy, interference,reportable range and others. The CLIA 88 totalfixed error limits now provide definedperformance standards against whichanalytical methods can be assessed. Meetingor exceeding acceptable performancestandards for a method in large measure
accuracy can be objectively compared withexisting procedures or other methods underconsideration. The ultimate goal being thattotal analytical error should be reduced witheach succeeding methodology. Methodsevaluation should be integrated in to routinelaboratory operations as a mechanism topromote continual quality improvement.
Decision Making in the Clinical Laboratory Robert L. Klick
(include label cell), to select the block ofdata; click on Tools from the menu bar,then select Data Analysis from the dropdown menu (see Figure 9). (If DataAnalysis does not show up on this menu,check your Add-Ins. The module must beinstalled before using.)
The next dialog box ( Figure 10) lists all thedata analysis packages available. To obtainbasic statistical parameters, click onDescriptive Statistics, then the OK button.
Decision Making in the Clinical Laboratory Robert L. Klick
The next dialog box ( Figure 11) allowsthe user to specify the Input Range (inthis case, A1:A16); the “Labels in FirstRow” is checked, since the first row ofour selection contains the column labels. The output (calculations) will be placedin the cells beginning with B1 (ie, B1 isthe upper, left-hand cell of the outputblock). Also note that “SummaryStatistics” is checked to provideadditional statistical information.
Clicking the OK button performs thecalculations and places the results in thedesignated area of the spreadsheet. Toview all labels and numbers completely,click Format >> Column >> AutoFitSelecton (Figure 12).
Finally, the %CV value is easilycalculated by entering the appropriateformula, (C7/C3)*100, in an unusedcell; in this case, B18 (see Figure 13).
As noted in the presentation, clinicallaboratory method comparisons areoften better evaluated using Demingsregression analysis, which takes intoaccount the measurement error in bothmethods. As a special bonus, thisMed TechNet presentation is beingdistributed with an Excel spreadsheetthat will perform the Demingscalculations. Be sure to download thefile, MTCon19.xls, available with thisthe presentation file ( mtc19pdf.pdf).
Additional instructions for using thetemplate are found on the Instructionsworksheet. MTCon19.xls will work withMicrosoft’s Excel version 5 and higher.
Decision Making in the Clinical Laboratory Robert L. Klick
1. Clinical Laboratory Improvement Act; FederalRegister, Feb 28, 1992, vol. 57 (40) : 7149-7184and Federal Register, Jan 19, 1993, vol. 58(11):5230-5232.
2. Comprehensive Accreditation Manual forPathology and Clinical Services, 1996. JointCommission on Accreditation of Health CareOrganization, Oakbrook Terrace, Illinois
3. Fraser, CG. Analytical Goals are Applicable toAll. J Int Fed Clin Chem. 1990; 2(2): 84-86.
4. Koch, DD and Peters , Jr, T. Selection andEvaluation of Methods: With an Introduction ofStatistical Techniques. In Tietz, Fundamentalsof Clinical Chemistry, 4th Ed.,1996: 170-181.
Editors Burtis, CA and Ashwood, ER, W.B.Saunders Co., Philadelphia
6. Ehrmeyer, SS, Laessig, RH, Leinweber JE andOryall, JJ. 1990 Medicare/CLIA Final Rules ofProficiency Testing: Minimum InterlaboratoryPerformance Characteristics (CV and Bias)Needed to Pass. Clin Chem. 1990;36(10):1736-1740
7. CLIA Watch, Guide to Implementing CLIARegulations on the Laboratory: Calibration,Calibration Verification and Other VerificationProcedures. Sigma Chemical Company, March,1993.
8. Cary, RN, Garber, CC and Koch, DD. Conceptsand Practices in the Evaluation of LaboratoryMethods. Workshop #308, Am Assoc ClinChem, 1993, New York: 1-97.
9. National Committee for Clinical LaboratoryStandards, EP7 Interference Testing.6(13):259-366.
Med TechNet is a service of Western New York Microcomputer, Inc.PO Box 84
East Amherst, NY 14051, USA
Med TechNet Online Presentations
The online discussion for this presentation can be found in a dedicated Message Area, onthe Med TechNet BBS; or, may be accessed by subscribing to the Internet list for thispresentation (see below).
Reaching the Med TechNet BBS
The Med TechNet BBS is a menu driven online service with many special and generalizedconferences, as well as a complete online software library with thousands of titles, forDOS, Windows 3.1x, Windows 95/NT and OS/2. Users can explore the BBS by telnet’ing toMed TechNet on the Internet at, bbs.medtechnet.com, or dialing in directly, via modem(V.34), at, 1-716-688-1552. (For more information on “telnet”, see,http://www.medtechnet.com/telnet.html.)
Online Discussions
Discussions for Med TechNet presentations are available with the author(s) and otherpresentation participants. Either telnet or dial-in to the BBS and go to the appropriatediscussion area in the MESSAGE AREAS. Or, if you prefer, participate via Internet E-Mail. Complete instructions for joining the Internet list for this presentation are available tosubscribers who register online at the Med TechNet web site.
Earning CEU’s
Earning CEU’s for this presentation requires a passing score on the post-quiz. Successfulcompletion of the online, electronically-scored post-quiz earns the participant 4.0P.A.C.E.® Contact Hours or California State accredited credits.
Registration and quizzes are administered at the Med TechNet web site(http://www.medtechnet.com/). You must be a Med TechNet subscriber to register forCEU’s. You must register for a presentation to access the post-quiz.
![Clin Ind 4[v6.0]](https://static.documents.pub/doc/80x56/577cc7971a28aba711a16928/clin-ind-4v60.jpg)
