Evaluation of Image Quality of MRI Data for Brain TumorSurgery

Frank Heckela,b, Felix Arltc, Benjamin Geislera, Stephan Zidowitza, Thomas Neumuthb

aFraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany;bInnovation Center Computer Assisted Surgery, Leipzig, Germany;

cLeipzig University Hospital, Department of Neurosurgery, Leipzig, Germany

ABSTRACT

3D medical images are important components of modern medicine. Their usefulness for the physician dependson their quality, though. Only high-quality images allow accurate and reproducible diagnosis and appropriatesupport during treatment. We have analyzed 202 MRI images for brain tumor surgery in a retrospective study.Both an experienced neurosurgeon and an experienced neuroradiologist rated each available image with respectto its role in the clinical workflow, its suitability for this specific role, various image quality characteristics, andimaging artifacts. Our results show that MRI data acquired for brain tumor surgery does not always fulfill therequired quality standards and that there is a significant disagreement between the surgeon and the radiologist,with the surgeon being more critical. Noise, resolution, as well as the coverage of anatomical structures were themost important criteria for the surgeon, while the radiologist was mainly disturbed by motion artifacts.

Keywords: image quality assessment, magnetic resonance imaging, brain tumor surgery

1. INTRODUCTION

3D medical images acquired by magnetic resonance imaging (MRI) for example are ubiquitous in today’s surgery.They are used in each stage of the clinical workflow for diagnosis, therapy planning, image-guided intervention(e.g., navigated and minimally invasive surgery), and therapy success monitoring. The mere technical availabilityof the data, however, does not necessarily ensure that the required information is provided. The quality of animage, for instance, is one of its most important relevance criteria.1 The requirements on an image’s qualitydepend on the specific clinical use-case. If they are not fulfilled, the data cannot be used, it is of limited value, orit could even lead to inaccurate diagnosis.2,3 Particularly computer-assisted systems, such as automated imageanalysis or intra-operative navigation, are known to need high quality images in order to provide accurate results.4

This study evaluates the suitability of MRI data in the context of brain tumor surgery. In brain tumor surgery,the extent of surgical resection is linked to patient outcome. On the one hand, all tumor cells need to be removedin order to avoid recurrence. On the other hand, healthy brain tissue and critical risk structures, such as thecerebral cortex, motor, somatosensory, language, and visual function areas, have to be protected from damage.5,6

Therefore, high quality images are particularly important for accurate therapy planning and intra-operativenavigation. We have analyzed the influence of image quality on the suitability of the images for their specificpurpose as well as the agreement between the surgeon and the radiologist.

2. MATERIAL AND METHODS

We have analyzed MRI data from 24 patients in a retrospective study. For each patient, different images (sequencesand reconstructions) were available, resulting in a total number of 202 datasets. The data was acquired at LeipzigUniversity Hospital for brain tumor surgery at the Department of Neurosurgery. Both an experienced neurosurgeonand an experienced neuroradiologist have annotated each image in a visual grading analysis using an absolutegrading on an adjectival scale.7

The annotation has been done using an intuitive, flexible tool that has been developed using MeVisLab∗ (seeFig. 1). The ratings and comments as well as meta-information for each image have been stored in an XML file

∗http://www.mevislab.de

Figure 1: The image quality annotation tool that has been used for visual grading analysis.

for further analysis. The following information has been collected: the purpose (“role”) of the image (diagnosis,therapy planning, intra-operative navigation) and its overall suitability for this specific role; the coverage ofthe overall anatomical region (i.e., the brain; we refer to this as “data coverage”) and the target region (i.e.,the specific tumor); general image quality characteristics (signal-to-noise ratio, resolution, homogeneity); andMRI-specific imaging artifacts (motion, ghosting, susceptibility). Images could also be rated as irrelevant. Thiswas done for instance for functional MRI sequences, diffusion weighted images, and sequences without contrastagent, which are not commonly used by surgeons in any of the workflow steps. In addition, the clinicians wereallowed to skip images if they were not able to rate them appropriately.

The ratings have been statistically analyzed. Specific quality aspects have also been correlated with the overallsuitability. Spearman’s rank correlation coefficient has been used to measure correlation. Finally, the agreementbetween the surgeon and the radiologist has been analyzed. The overall agreement is computed by

d =1

n

∑i

‖ai − bi‖, (1)

with ai and bi being the annotation by reader 1 and reader 2 mapped to integers ([0,4] for coverage, signal-to-noiseratio, resolution, and homogeneity; [0,3] for MRI-specific artifacts). Wilcoxon’s signed-rank test has been used tomeasure the significance of the agreement. The significance level was set to α = 0.05.

Figure 2: Role and suitability as rated by the neurosurgeon and the neuroradiologist.

3. RESULTS

Figure 2 shows an overview on the different roles of the images and their overall suitability as rated by theneurosurgeon and the neuroradiologist. Obviously, only the surgeon can judge whether a specific image has beenacquired for intra-operative navigation, while the radiologist judges the images mostly from a diagnostic pointof view. For a surprisingly high number of images the overall suitability has been rated as bad or unusable,particularly by the surgeon, who also seems to be more critical concerning the suitability of the images. Theradiologist skipped less images as he also considered images without contrast agent.

From the surgeon’s point of view, the coverage of both the whole anatomical region and the target region wasnot always sufficient, i.e., important parts were missing in the images (see Fig. 3). This correlates well with thesuitability rating. A relatively high number of images shows a bad resolution, signal-to-noise ratio, or homogeneity(see Fig. 4). Again, the surgeon gives worse ratings compared to the radiologist. For the surgeon, the generalquality characteristics correlate well with the suitability. For the radiologist, the only parameter that shows agood correlation with the suitability is the presence of motion artifacts (see Fig. 5). Overall, the radiologist ismore critical with respect to imaging artifacts compared to the surgeon.

For suitability, coverage, and the general quality characteristics, the surgeon and the radiologist show a ratherbad agreement on average (see Figs. 2 - 5). Only for imaging artifacts the agreement is relatively good. Note,however, that disturbing imaging artifacts are rather seldom and most images artifacts were rated as “not present”.The result for the agreement are significant for all evaluated parameters. Note that the evaluation of agreementonly considers cases that have been rated by both readers.

Figure 3: Influence of coverage on the suitability. ρ refers to Spearman’s correlation coefficient.

4. DISCUSSION

The assessment of the quality of natural and compressed images as well as videos has been a topic of intenseresearch in the past decades.8 In this context, quality is often defined in a subjective manner as perceived by ahuman observer. Consequently, there have been many attempts to model human observers for automatic qualityassessment with respect to specific tasks.8,9 In medical imaging, image quality assessment is an emerging fieldwhere most research focuses on image quality control during the acquisition.4,10–13 In recent years, however,researchers also started modeling human observers.7,14

In this study, we have analyzed the suitability of medical images for their specific purpose and the influence ofimage quality on their overall suitability from a clinical point of view based on a specific clinical use-case. Invarious interviews, physicians stated that “images are almost always good and their quality is not a problem inpractice”. In contrast to this subjective impression, our results indicate that there is a relevant number of imagesthat do not meet the required quality standards for the examined clinical use-case. The results also suggest thatthose standards show a significant gap between radiologists and surgeons, with surgeons being more critical.Coverage and general image quality characteristics, such as resolution and signal-to-noise ratio for example, seemto be the most relevant aspects from the surgeon’s point of view.

The main limitation of this study is the small number of clinical participants with only one representativefrom each discipline. In addition, the images have been acquired from only one site, which is a university hospitalwith modern equipment and high standards. As absolute grading studies are known to have a low reproducibility,more readers with different levels of experience have to be included in future studies. Alternatively, relativegrading might be used.7

The results of our study can act as baseline data for automatic quality assessment tools for neurosurgicalinterventions. Building automatic image quality assessment algorithms requires either perfect reference imagesto which new images are compared (full-reference methods) or it is based on the detection of specific types of

Figure 4: Influence of general image quality characteristics on the suitability and agreement between disciplines. ρrefers to Spearman’s correlation coefficient.

distortion, such as blurring or noise (no-reference methods).8 As reference images are not available for mostimages acquired in medical imaging, no-reference quality assessment methods are required.

5. CONCLUSION AND FUTURE WORK

This study analyzed the suitability MRI data in the context of brain tumor surgery from a clinical point ofview. We have evaluated the influence of image quality and imaging artifacts on the overall suitability and wehave compared the results of a surgeon to the results of a radiologist. Our results show that MRI data acquiredfor brain tumor surgery does not always fulfill the required quality standards and that there is a significantdisagreement between the surgeon and the radiologist.

Figure 5: Influence of the most relevant MRI-specific artifacts on the suitability and agreement between disciplines.ρ refers to Spearman’s correlation coefficient.

Further studies with different clinical use-cases, more data from divergent radiology practices, and moreexperts are necessary to generalize our results. Our future work will also focus on no-reference automatic imagequality assessment in order to automatically detect unsuitable image as early as possible in the clinical workflow,as it has previously been proposed by Woodard & Carley-Spencer,15 Schwaab et al.,16 and Kustern et al.17 forexample.

ACKNOWLEDGMENTS

This work was funded by the German Ministry of Education and Research (BMBF) as part of the project OR.NET(reference number: 16KT1232).

REFERENCES

[1] Sedghi, S., Sanderson, M., and Clough, P., “A study on the relevance criteria for medical images,” PatternRecognition Letters 29(15), 2046–2057 (2008).

[2] de Certaines, J. D. and Cathelineau, G., “Safety aspects and quality assessment in MRI and MRS: A challengefor health care systems in Europe,” Journal of Magnetic Resonance Imaging 13(4), 632–638 (2001).

[3] Mortamet, B., Bernstein, M. A., Jack, C. R., Gunter, J. L., Ward, C., Britson, P. J., Meuli, R., Thiran, J.-P.,Krueger, G., and Alzheimer’s Disease Neuroimaging Initiative, “Automatic quality assessment in structuralbrain magnetic resonance imaging,” Magnetic Resonance in Medicine 62(2), 365–372 (2009).

[4] Gedamu, E. L., Collins, D. L., and Arnold, D. L., “Automated quality control of brain MR images,” Journalof Magnetic Resonance Imaging 28(2), 308–319 (2008).

[5] Mastronardi, L., Bozzao, A., D’Andrea, G., Romano, A., Caroli, M., Cipriani, V., Ferrante, M., and Ferrante,L., “Use of preoperative and intraoperative magnetic resonance tractography in intracranial tumor surgery,”Clinical Neurosurgery 55, 160–164 (2008).

[6] Orringer, D. A. and Jolesz, F. A., “A rationale for the use and development of methods for image-guidedbrain tumor surgery,” in [Intraoperative Imaging and Image-Guided Therapy ], 479–483, Springer New York(2014).

[7] Verdun, F. R., Racine, D., Ott, J. G., Tapiovaara, M. J., Toroi, P., Bochud, F. O., Veldkamp, W. J. H.,Schegerer, A., Bouwman, R. W., Giron, I. H., Marshall, N. W., and Edyvean, S., “Image quality in CT: Fromphysical measurements to model observers,” Physica Medica 31(8), 823–843 (2015). online first 10/2015.

[8] Chandler, D. M., “Seven challenges in image quality assessment: Past, present, and future research,” ISRNSignal Processing 2013, 53 (2013). Article ID 905685.

[9] Barrett, H. H., Yao, J., Rolland, J. P., and Myers, K. J., “Model observers for assessment of image quality,”Proceedings of the National Academy of Sciences 90(21), 9758–9765 (1993).

[10] Ihalainen, T., Sipila, O., and Savolainen, S., “MRI quality control: six imagers studied using eleven unifiedimage quality parameters,” European Radiology 14(10), 1859–1865 (2004).

[11] Friedman, L. and Glover, G. H., “Report on a multicenter fMRI quality assurance protocol,” Journal ofMagnetic Resonance Imaging 23(6), 827–839 (2006).

[12] Fagiolo, G., Hajnal, J. V., Hill, D., Fox, N., Kennedy, J., Mcleish, K., Waldman, A., and Dresner, A., “QABox:Automatic real-time MR image quality assurance system for clinical trials,” in [Proceedings of ISMRM ], 1886(2007).

[13] Tsai, D.-Y. and Matsuyama, E., “Recent advances of quality assessment for medical imaging systems andmedical images,” in [Visual Signal Quality Assessment ], Deng, C., Ma, L., Lin, W., and Ngan, K. N., eds.,157–183, Springer International Publishing (2015).

[14] He, X. and Park, S., “Model observers in medical imaging research,” Theranostics 3(10), 774–786 (2013).

[15] Woodard, J. P. and Carley-Spencer, M. P., “No-reference image quality metrics for structural MRI,”Neuroinformatics 4(3), 243–262 (2006).

[16] Schwaab, J., Diez, Y., Martı, J. M. R., van Zelst, J., Tan, B. P. T., Gregori, J., Wirtz, S., Kramme, J., andGunther, M., “Image quality in automated breast ultrasound images: a preliminary study for the developmentof automated image quality assessment,” in [Proceedings of MICCAI International Workshop on BreastImage Analysis ], 122–129 (2013).

[17] Kustner, T., Bahar, P., Wurslin, C., Gatidis, S., Martirosian, P., Schwenzer, N. F., Schmidt, H., and Yang,B., “A new approach for automatic image quality assessment,” in [Proceedings of ISMRM ], 2493 (2015).