The IVD Directive andAvailability of ReferenceSystems for IVD MedicalDevices: A View fromIndustry

Neil Greenberg, PhD, DABCC, Manager,Regulatory Affairs, Ortho-ClinicalDiagnostics, Inc., A Johnson & JohnsonCompany

This article is published with permis-sion and originally appeared:Neil Greenberg, “Calibrator Traceability:The Industry Impact of the IVD Direc-tive’s New Requirements,” IVD Technol-ogy 7, no. 2 (2001): 18-27

Introduction

Traceability to internationally recognized andaccepted standards is an important component inassuring the accuracy and comparability of clinicallaboratory measurements. Currently, the globalmarketplace is presenting new demands on IVDdevice manufacturers for measurement traceability.Under a mandate from the European Union’s InVitro Diagnostics Directive1 (IVDD), the EuropeanCommittee for Standardization’s Technical Commit-tee 140 (CEN/TC 140 - In Vitro Diagnostic Sys-tems), together with ISO/TC 212, is currentlydeveloping international standards on IVD calibra-tion traceability2,3. Full implementation of the IVDDirective, which is obligatory by December 2003under European law, will require that calibration ofquantitative IVD assays be traceable to available“higher-order” reference materials or methods.Manufacturers who implement the processes anddocumentation steps defined in the calibrationtraceability standards are entitled to a presumptionof conformity to this “essential” requirement of theIVD Directive. To do this, IVD manufacturers mustensure that the systems they market have beencalibrated against available higher-order referencestandards and procedures, that repeatability andreproducibility of their internal calibration proce-dures are quantified and documented, and thataccuracy is substantiated by uncertainty calculations.

The draft international standard for IVD calibrationtraceability, ISO/DIS 17511, identifies the essentialelements of a calibration hierarchy necessary tosupport full calibration traceability to defined unitsof measure under the Système International (e.g.moles, kilograms). In addition to a completedefinition of the quantity to be measured, there is aneed for primary standards, including materialstandards as well as standard methods of measure-ment. In the introductory section of the text, thestandard also discusses a key problem that exists formeasurement systems in the field of in vitrodiagnostics. This is that, although the in vitrodiagnostics field routinely performs measurementson an estimated 400 to 600 different amounts ofsubstances (analytes), full calibration systems withtraceability to SI currently exist for less than 30(perhaps 5%) of these analytes. What will be doneto fill this 95% void, and what role will be assumedby the IVD industry?

Do IVD manufacturers want additionalinternational reference materials andmethods to be developed?

Despite short-term concerns, international IVDcalibration standards (reference systems) ultimatelyhelp IVD manufacturers by providing well-definedmarket needs and customer requirements, a clear anduniversal definition of goals, and tools for objectiveassessment of product attributes. Table 1 lists a fewof the global trade-offs to be evaluated by allparticipating organizations (manufacturers anddistributors of IVD products, industry associations,user groups, customer advocacy and professionalgroups, government and regulatory bodies) uponconsidering development of new calibrationstandards (reference systems) for IVDs. It is clearly amatter of cost vs. benefit, and the decision to investin standardization demands careful analysis on acase-by-case basis.

An Example: Economic Impact Analysis ofStandard Reference Materials forCholesterol:

A recent economic impact study conducted by the USNational Institute of Standards and Technology(NIST) quantified a portion of the economicbenefits associated with the availability of NISTCholesterol Standard Reference Materials beginningin 19864. This study determined that the economic

Page 80eJIFCC2001Vol13No3pp080-084

consequences of NIST’s Cholesterol StandardsProgram were experienced at several levels of theIVD medical device supply chain from manufactur-ers, to network laboratories, and to clinical laborato-ries that ultimately deliver medical services to theconsumer. The nature of the benefits to industryresulting from the NIST investment have changedover more than three decades of NIST involvement.However, because the timeframe of this analysis waslimited to 1986-1999, only part of the NISTprogram’s life cycle, the magnitude of the estimatedeconomic impact was biased low. Nevertheless, theresults indicated that NIST reference materials playedan important economic role in support of the USnational effort to monitor, measure, and controlcholesterol levels, thereby contributing to a reducedlevel of heart disease. The study estimated a benefit-to-cost ratio of 4.5, and a social rate of return of 154%. The Net Present Value was calculated to be morethan (US) $3.6 million. The study did not attemptto account for the impact of NIST reference materi-als on reducing the incidence of incorrect laboratorymeasurements on patient care, which has beenestimated to be approximately $100 million per yearin the US.

Impact of Measurement Bias onGlobalization of Healthcare

In the 21st century, we live in an era where patientsfrequently move from town to town or even countryto country as often as every few years. Additionally,for economic reasons, there is an increase in the rateat which patients change to different health careplans and different clinical laboratories. As themobility of medical records increases along with

patient mobility, differences in laboratory test resultsamong laboratories and across different test meth-ods are becoming increasingly apparent to physiciansand other health care practitioners.

Problems due to lack of consistency in standardiza-tion among analytical methods were highlighted andpublicly debated in the United States during the1980’s, following the US National Institutes ofHealth publication of its findings on the relation-ship between serum cholesterol levels and risk forcardiovascular disease. Similarly, in the mid-1990’s, arecommendation was published for monitoring thechange in serum PSA over 2 to 4 years, as an aid inthe diagnosis of prostate cancer. This recommenda-tion made inter-laboratory variability in PSAdeterminations highly problematic, since a largechange in PSA could result simply because ofanalytical differences in the methods used by twolabs. Alternatively, clinically significant changes in truePSA levels could be masked for the same reasons.

Klee5 has evaluated the cost-impact of certain biasedlaboratory test results associated with unnecessaryfollowup of mis-classified patients. For a screeningtest such as serum cholesterol, patient mis-classifica-tion is likely to be followed up with expensiveadditional testing or even inappropriate treatment.Using actual test results distributions rather thanGaussian models, Klee estimated that an assay forserum cholesterol that is biased 1.0% high results ina 3.0 % increase in the number of patients classifiedas having “high” cholesterol values. Similarly, a 3 %bias causes an 8.8 % increase in the number ofpatients classified as hypercholesterolemic, and a 10% bias causes a 27.8 % increase. Overall, the changein the percentage of patients crossing the serumcholesterol decision threshold (5.17 mmol/L; 200

Interchangeability of data between products

Competitiveness – level playing field for competition

Defined quality goals followed by evolution of products toward the goal

Lower long-term costs

Clearer pathway to market access

Transferable technology

Independent tools to ensure long-term performance stability

Diverting qualified people to participate in standards work (vs. other programs)

Risk of investing in standards that, upon completion, are not accepted by all stakeholders, especially customers

Lengthy cycle time to achieve deliverables

Costs of transition (both within manufacturing companies & for customers) to make changes to comply with new standardsLess variety; fewer alternatives for customers

Barriers to innovation

Barriers to market entry

BENEFITS

COSTS

TABLE 1. Decision Trade-Offs for Investment in New IVD Reference Systems

Page 81eJIFCC2001Vol13No3pp080-084

mg/dL) increases about three fold more than thepercentage value of the analytical shift due to themultiplier effect of the distribution curves. Based onthese estimates, improvements in laboratorystandardization (especially for screening tests forclinical conditions with high prevalence) makessound economic sense, whenever the outcome islikely to be lower rates of patient mis-classification.

Key factors leading to successfulimprovement in inter-laboratorystandardization

Why do some standardization programs meet withsuccess, while other programs seem to languish insome instances for years without yielding noticeablebenefits? Following up on the examples of choles-terol and PSA discussed above, it is clear that theprograms to improve standardization of serumcholesterol measurements yielded measureablesuccess in terms of a very substantial reduction inmeasurement variability among laboratories over aten year period. However for PSA, although someprogress is evident, success remains elusive6.Similarly, inter-method and inter-laboratory stand-ardization efforts for measurement of humanchorionic gonadotropin (HCG) in serum and urine,another clinically important analyte which is anexcellent marker for screening and monitoringnormal and abnormal pregnancy as well as certainmalignancies, have also encountered significanttechnical barriers7.

In a recent review of the history of standardizationefforts in the clinical laboratory, Eckfeldt8 identifiedfour significant antecedents to successful laboratorystandardization programs. These include:

� Results of a widely publicized clinical researchstudy conclude that clinical action based onapplication of uniform cut-points for a particularlaboratory test leads to significant improvementin detection and prognosis of patients withdisease. This new information leads to pressurefrom large clinical organizations for improve-ments in test method accuracy and reliability.

� A high-level reference method and/or materialexists.

� Mechanisms exist to easily and reliably dissemi-nate the accuracy base provided by the referencemethod and/or material.

� Tools exist to reliably evaluate and publiclydisplay inter-method and inter-laboratoryperformance data. EQAS programs are a primarysource of such information, and the value ofsuch programs depends on the free flow ofinformation about their procedures and the testmaterials that they distribute.

The importance of the reliability of EQAS data inassessment of the accuracy of laboratory methods isoften underestimated. It is essential that EQASmaterials achieve the highest possible level ofcommutability, so as to ensure their validity inrepresentation of performance of a given laboratorytest with patient samples. Indeed, incorrect assump-tions about the validity and commutability ofEQAS materials may lead to incorrect conclusionsand even serious errors on the part of IVD kitmanufacturers as well as EQAS providers andlaboratorians. In the 1980’s, before informationabout commutability problems and matrix effectswith manufactured (typically lyophilized) serumcontrols and EQAS/PT materials was widelypublicized, there were examples where manufacturersunwittingly adjusted the calibration of their IVDdevices and reagents to make PT/EQAS samples’results comparable to reference method target values.In some cases, these adjustments compromisedaccuracy with patient samples.

Concerns about the suitability of PT/EQASmaterials have led to uncertainty regarding the valueof PT/EQAS results in understanding the state-of-the-art for trueness. These concerns have also beenresponsible for creating some tension betweenproviders of PT/EQAS programs and IVDmanufacturers whose commercial methods areevaluated by these programs. IVD manufacturersoften respond that inferior PT materials do notaccurately simulate clinical specimens, and mis-represent the performance of their methods. Theproviders of PT/EQAS materials and programsoften argue that commercial reagent and instrumentsystems are insufficiently “robust”. In reality, the“problem” must be shared by the broader profes-sional laboratory community and the commercialIVD industry as a whole.

Labeling changes and product inventory obsolescence,Customer and EQAS program advisory noticesFilings with regulatory authoritiesManufacturing process changesTraining of customers and intra-company personnelReference interval updates requiring additional communications and training for clinicians

Short-term IVD Manufacturer & Laboratory Costs of Calibration Changes

Table 2

Page 82eJIFCC2001Vol13No3pp080-084

Declaration of New or ImprovedInternational Reference Materials andReference

Methods – Some Manufacturers May Needto Change Calibrations for CertainAnalytes

Development of new or improved reference systemsis not a specific requirement of the IVD Directive.The Directive states that routine methods need to betraceable to “…available reference measurementprocedures and/or available reference materials of ahigher order.” Similarly, ISO/CD 17511 allows for awide range of scenarios, including situations wherethere is no recognized higher order calibrationmethod or material available to trace back to. Underthese circumstances, the highest order reference pointavailable to a given IVD manufacturer may be ameasurement procedure or reference material that isuniquely defined, controlled, and maintained by eachmanufacturer of the various commercial assaysystems for a given analyte. ISO/CD 17511 takes astronger position in terms of commitment to thecause of continuous improvement in referencemethods and materials. As the standard states, “It isthe aim of metrology in laboratory medicine toimprove traceability…by providing the missingreference measurement procedures and referencematerials, based on international consensus.”Because of this implied commitment, it is expectedthat many national and international standardsorganizations, scientific, professional, and industrygroups, will interpret the Directive’s intent, arrivingat an interpretation which says that the Directivedemands investment in upgrades to the interna-tional reference system for the clinical lab.

When new reference methods or materials aredeveloped and become globally accredited, certainIVD manufacturers will inevitably have to makechanges in their internal calibration procedures inorder to adjust performance of their products tobecome standardized to new reference systems9. Thecost implications for these changes are far-reaching,and will impact end-users as well. Table 2 highlightsa few of these costs.

Most successful IVD device manufacturers arewilling to step up and implement calibration changesnecessitated by customer needs, especially when thesechanges are expected to be beneficial in bringingabout improved health care. Hopefully, the requiredchanges will be important ones, representing needsarticulated by a broad, global consensus of cliniciansand laboratory professionals, since many changesmay be associated with disruption and costs. Giventhese costs, it is especially important that theinitiative and the leadership for change be customer-focused, originating from a clear expression of needfor improvement on the part of the end-users, notthe manufacturers.

What is most important, as new reference materialsor reference method projects are initiated in the nameof the Directive, is that adequate scientific supportfrom industry be sought when staffing the technicalworking groups. Active and meaningful participationby industry scientists will help to ensure that areasonable balance is achieved between commercialinterests, pragmatic realities of manufacturingmaterials and process limitations, and academicclinical and metrological interests.

Does Industry Support Initiatives for NewIVD Calibration and Reference Systems?

Public interest requires good quality and safe healthcare products. It is easy to demonstrate that im-proved standardization contributes to furtheringthese goals, and all IVD manufacturing companiesshare this interest. Reference materials and referencemethods for calibration are an important underlyingelement, contributing to quality and safety throughthe assurance of interchangeability of informationacross time and space.

Given these shared goals, it is inevitable that newreference systems projects will emerge, and newreference materials and methods will ultimately bedefined, impacting the definition of the state of theart. Industry must play a role in this process, usingits collective wisdom achieved through years of real-world experience, to ensure technically sound andpractical solutions to the challenges encountered indevelopment projects undertaken in the quest forbetter standards.

Recommendations

What factors are necessary to get to the future state?To begin, laboratory medicine, scientific, andprofessional organizations need to provide leader-ship and guidance relative to what standards areneeded. In doing so, project priorities should bedefined with an appreciation that resources arelimited, while taking into account factors such as (1)the public health significance and disease course, (2)expectations of the degree of improvement antici-pated in overall clinical effectiveness of a given test ifa new standard is developed, (3) time and costestimated to reach a desirable endpoint, and (4) theoverall likelihood of success.

High priority should be given to establishing adefined, global, customer-focused, and consensus-based process for setting priorities and contractingprojects. This process should be led by a consortiumof the world’s major laboratory professionalassociations (e.g. IFCC, AACC, CAP, WASP andothers) and should proceed in an atmosphere ofopen and public dialogue, employing decision toolsthat emphasize quantification and metrics, and isinclusive of all key stakeholders (i.e., profession,government, industry, lay public).

Page 83eJIFCC2001Vol13No3pp080-084

Sufficient project funding (grants or contracts) mustbe provided, and should include governmentsources (e.g. EU Commission, US Department ofCommerce, etc.), professional societies (e.g. IFCC,AACC, CAP, WASP), as well as industry groups.Projects must be closely managed, with accountabil-ity for deliverables and schedule. Project teams mustbe staffed with appropriate clinical and scientificexperts, coming equally from the professions andindustry.

Conclusions

Although the EU IVD Directive does not specificallyrequire it, reference materials and reference methodsdevelopment projects will be initiated in the nameof the Directive’s essential requirement for calibra-tion traceability. New calibration standards are likelyto increase short-term costs for IVD manufacturers,but this is a minor consideration if there is a clearneed for improved standards and an expectation ofimproved quality of health care as an outcome, asexpressed by a consensus of customers and profes-sional associations.

Professional and customer advocacy groups shouldtake the lead role in advocating for new calibrationstandards, especially in defining where improvedstandards are needed. Project selection must utilizecost-benefit analysis, taking into account publichealth payback, technological limitations, andmagnitude of the investment necessary to achievethe desired outcome.

Whenever IVD calibration and reference systemsstandards projects are undertaken, whether spon-sored by professional, government, or public healthgroups, inclusion of IVD industry scientists andexperts on the technical team is an absolute prerequi-site for project success. Creative strategies are neededto ensure adequate project funding, and shouldinvolve a combination of contributions from public,professional and industry sources.

References

1 Council Directive 98/79/EC of the EuropeanParliament and of the Council of 27 October1998 on In Vitro Diagnostic Medical Devices,Official Journal of the European Union L331(December 12, 1998).

2 In Vitro Diagnostic Medical Devices—Measure-ment of Quantities in Samples of BiologicalOrigin—Metrological Traceability of ValuesAssigned to Calibrators and Control Materials,ISO/CD 17511 (Geneva: International Organiza-tion for Standardization, February, 2000).

3 In Vitro Diagnostic Medical Devices—Measure-ment of Quantities in Samples of BiologicalOrigin—Metrological Traceability of Values forCatalytic Concentration of Enzymes Assigned toCalibrators and Control Materials, ISO/CD18153 (Geneva: International Organization forStandardization, February, 2000).

4 May, Willie E, NIST Measurement Methods andStandards Reference Materials for Health StatusMarkers: Current Program and Future Challenges,in: NISTIR 6742, Proceedings of the Workshopon Measurement Traceability for Clinical Labora-tory Testing and In Vitro Diagnostic TestSystems, Ellyn S. Beary, Editor, May 2001, U. S.Department Of Commerce, National Instituteof Standards and Technology.

5 Klee, George G., Importance of CommutableReference Materials and Patient Test Distributionsfor Assay Calibration, in: NISTIR 6742, Proceed-ings of the Workshop on Measurement Trace-ability for Clinical Laboratory Testing and InVitro Diagnostic Test Systems, Ellyn S. Beary,Editor, May 2001, U. S. Department Of Com-merce, National Institute of Standards andTechnology.

6 Chan DW, Sokoll LJ. WHO first internationalstandard for prostate-specific antigen: thebeginning of the end for assay discrepancies? ClinChem 2000;46:1291-2.

7 Galina Kovalevskaya, Steven Birken, TatsuKakuma, John Schlatterer and John F.O’Connor, Evaluation of Nicked HumanChorionic Gonadotropin Content in ClinicalSpecimens by a Specific Immunometric Assay,Clinical Chemistry, 1999; 45: 68-77.

8 Eckfeldt, John H., History of Reference Systemsfor Clinical Measurements, in: NISTIR 6742,Proceedings of the Workshop on MeasurementTraceability for Clinical Laboratory Testing and InVitro Diagnostic Test Systems, Ellyn S. Beary,Editor, May 2001, U. S. Department Of Com-merce, National Institute of Standards andTechnology.

9 Powers, Donald M., Regulations and Standards -Traceability of assay calibrators: The EU’s IVDDirective raises the bar, IVD Technology, July-August, 2000.

Page 84eJIFCC2001Vol13No3pp080-084