Principles of Software Verification and Validation for Medical Imaging
Twin SpinOctober 7, 2010
07-OCT-2010
Topics
2
Short history of medical imaging How SW “Changed the Picture” Challenges and Solutions to Software
Verification 4 Areas of image trustworthiness Importance of SW Engineering Principles
Product Software Validation Summary
07-OCT-2010
History
07-OCT-2010 3
Röntgen discovers X-rays in 1885, receives Nobel Prize in 1901.
Rapid research and discovery leading to working prototype x-ray imaging system by Edison in 1901
Other modalities follow rapid development in 20th century.
12/10/2008 4
Boom in x-ray-associated marketing.
Medical Imaging Today
Multiple Modalities Electron microscopy Radiographic – standard x-rays, fluoroscopy MRI Nuclear medicine – PET, gamma Thermography Tomography – CT scans Ultrasound Photoacoustic imaging - lasers + ultrasound.
12/10/2008 5
Medical Imaging Today
607-OCT-2010
Demonstration
2D images on e-film portable reader, courtesy my CT scan.
3D, 4D Images from Osirix open source reader. Courtesy
12/10/2008 7
For purposes of understanding these principles, this talk will focus on CT technology.
07-OCT-2010 8
How Software “Changed the Picture”Yesterday:
07-OCT-2010 9
Hardware config and control
Hardcopy output (film)
Manual archiving
Limited reviewHardware positioning
Today:
07-OCT-2010 10
Software config and control console
Software patient positioning control
Digital raw data
Dedicated digital signal processing
Transmission over enterprise network DICOM data output
Archival File Server (PACS) PACS reading station Advanced visualization workstation
The essential question:
Can I trust these pictures?
07-OCT-2010 11
To help answer the question, we ask two more questions:
Q: What is essential in the image?A: This is validation. “What is the right information?”
Q: How much error does the essential information contain?A: This is verification. “How do I insure the information is as error-free as possible?”
07-OCT-2010 12
The essential question changes:From:
Q: Can I trust these pictures?
To:Q: How much can I trust these pictures?
07-OCT-2010 13
Answering Question 1: An Art Lesson
Visual artists have known that images are an interpretation of reality and exploit that fact to convey essential messages.
Medical imaging is an interpretation of reality too, intentionally distorted through reconstruction algorithms to convey essential diagnostic information (i.e. right
information).
07-OCT-2010 14
Interpretation involves subjectivity.
Boy or girl?
How to create trust in a subjective process?
A: Experience
Through certification, advanced education, on-the-job training, etc.
07-OCT-2010 15
Product Validation
Validation involves the customer or it isn’t validation. Validation of images by medical imaging professionals is
absolutely vital: radiologists, CT/MR technologists, etc. Validation can be done at nearly all steps in the
development. Beta field phase absolutely essential. Some issues can only be validated
Human factors: presentation of on-screen information, segmentation preferences, image fidelity preferences.
Visualization of small vessels or structures whose phantoms are too costly.
12/10/2008 16
Product Validation Approaches
Internal panel of expert employees – training team, field application teams.
Internal panel of expert consultants – medical advisory board, focus groups.
Luminary sites willing to partner in product development.
Industry-acknowledged body of knowledge. Walter Reed colon datasets, Stanford bake-off.
12/10/2008 17
07-OCT-2010 18
User errorModality quantization,
noise errorReconstruction algorithmic error “false images”
Data transmission protocol error
Database error User error, visual errors 3D processing, user, & algorithmic errors
Answering Question 2: Dataflow error
Error in the dataflow.Type of Error Frequency Root source of error
User error at console Potentially often Human factors design, functional flaw.
Quantization and noise
Every scan Law of nature.
Reconstruction error Every scan Law of nature, functional flaw
Data transmission error
Rare and getting rarer
TCP/IP stack, DICOM stack, functional flaw.
Database error Rare and getting rarer
Configuration error, functional flaw.
PACS user error Potentially often Human factors design, image processing, functional flaw.
Viz workstation error Potentially often Human factors design, image processing, functional flaw.
07-OCT-2010 19
The technical effort simplifies to reducing the impact of undetected functional error on the image to be less than or equal to the impact of inherent error on the image SO THAT the images and data derived from the images are trustworthy.
Any “questionable” image abnormalities visual artifacts, or other deficiencies are due to laws of physics inherent in the modality.
07-OCT-2010 20
The four areas for an image to be trustworthy from the user perspective:
Image orientation Image fidelity Measurements Data integrity
07-OCT-2010 21
Workstation error in the end image.Area of concern Root source of error
Image orientation Functional error processing DICOM header and placing orientation markers on screen. Manifested by rotation of image by 90 degree increments on screen.
Image fidelity Functional error introduced by incorrect graphic engine processing and algorithm application. Manifested by obscured anatomy, rendering, incorrect segmentation.
Measurements Functional error processing scaling factor from DICOM header or error in measurement algorithms. Manifested by rulers wrong by large fixed multiples, measurement of same anatomical structures changing as code changes.
Data Integrity Failure in data transfer, processing of incoming DICOM data, cross-check of header information, management of database, volume. Manifested by missing or repeated areas in the 2D and 3D images.
07-OCT-2010 22
How to verify Image Orientation
07-OCT-2010 23
Scan multiple times
Manufacture an object of knowndimension, HU values, orientation
Check the result
The reference library of orientation datasets is ideal for regression automation.
Phantom manufacturers
http://www.universalmedicalinc.com/diagnostic-imaging/imaging-quality-control/phantoms/ct-phantom
http://www.phantomlab.com/rsvp_head.html http://www.cirsinc.com
12/10/2008 24
How to verify Image Fidelity
07-OCT-2010 25
Use reference datasets• Phantoms• Synthetic data
Check the result against reference image using image pixel checker
Problem: subtle differences due to video driver revs cause false failures. **Error below threshold of human eye** Tools must not flag errors that are not noticeable (or relevant) to the user. The sample image must be transferred through a daisy chain of imaging workstations to simulate the enterprise environment. Reprocessing the image can, in rare cases, lead to image degradation. User acceptance panels are another tactic – often for selecting default color tables.
DICOM xfer
How to verify Measurements
07-OCT-2010 26
Use reference datasets• Phantoms• Synthetic data
Check the result against reference measurement
DICOM xfer
In some cases it is necessary to manually complete a typical patient “workup” or workflow, transfer the patient record to a second workstation, and verify the measurements maintain consistency in data transfer & processing.
All manual measurements are, by their nature, subject to human error. Define +- bounds
How to verify Data Integrity
07-OCT-2010 27
Use reference datasets• Phantoms• Synthetic data•Large, small, •Multiple modalities•Load testing •Negative testing
Check the results at the database, not the user interface. We assume the user interface & visualization do not corrupt the data (it is prudent to verify this assumption if using this strategy).
DICOM xfer
DICOM Database
Data transfer testing is executed in isolation and in concert with orientation, fidelity, and measurement verification.
Other things that affect trustworthiness Reliability Stability Security Installation & Upgrades System integration Localization Licensing Performance Manufacturing & distribution Etc.
07-OCT-2010 28
In other words….
Very few of the verification factors for trustworthiness are medical imaging-specific. Most are core software engineering principles and good quality engineering. Good requirements development and management Good code development and management Good test case development and management
07-OCT-2010 29
Following good (not even best) practices automatically generates artifacts that audit agencies look for as proof of regulatory compliance.
Example of SW Engineering lapse.
GE Healthcare, August 2008.Optovue, 2010
12/10/2008 30
Risk-based verification
Everything can’t be tested Organizations inevitably take a risk-based approach
whether they know it or not. “Good enough” by instinct to begin “Good enough” by systematic classification by end.
Cross departmental effort to classify risks (patient risks, business risks)
Living document of decision-making as-you-go. Complies with regulatory risk assessment deliverables. High business value.
07-OCT-2010 31
Risk-based verification derived from Business Needs/Risks
All risks are secondary to patient risk. Recall vs. patch
07-OCT-2010 32
Business Need: (product will not
harm patient)
Project Risk Assessment(schedule,
market, patient hazards, etc)
Design Mitigation
Risk-based system testing
Unit testingBusiness Need:( )
Business Need( )
Summary
The key to medical imaging is trustworthiness of the image – even with known error sources.
Verifying medical imaging software is an engineering task approaching image orientation, image fidelity, measurements, and data integrity
Validating imaging software is a clinical user task overlapping with human factors and artistic questions of interpretation and presentation.
All other challenges are familiar core software engineering problems.
07-OCT-2010 33
Biography
Alex Dietz has been applying product quality verification principles for 20+ years in telecommunications, data transmission, and medical imaging. He currently manages the Software Verification and Validation team for the EnSite cardiac mapping system at St. Jude Medical. He has spoken locally and nationally, most recently at the Software Design for Medical Devices conference in San Diego.
07-OCT-2010 34
References
History Naked To The Bone: Medical Imaging In The Twentieth Century
by Bettyanne Kevles http://en.wikipedia.org/wiki/X-ray#History
Phantom Manufacturers http://www.phantomlab.com/ http://www.universalmedicalinc.com http://www.cirsinc.com
07-OCT-2010 35