Contents lists available at SciVerse ScienceDirect
Magnetic Resonance Imaging
j ourna l homepage: www.mr i journa l .com
False positive control of activated voxels in single fMRI analysis using bootstrapresampling in comparison to spatial smoothing
Fahimeh Darki, Mohammad Ali Oghabian⁎Neuro Imaging and Analysis group, Cell and Molecular Imaging Research Center, Tehran University of Medical Science, Tehran, Iran
Functional magnetic resonance imaging (fMRI) is an effective tool for the measurement of brain neuronalactivities. To date, several statistical methods have been proposed for analyzing fMRI datasets to select trueactive voxels among all the voxels appear to be positively activated. Finding a reliable and valid activationmap is very important and becomes more crucial in clinical and neurosurgical investigations of single fMRIdata, especially when pre-surgical planning requires accurate lateralization index as well as a preciselocalization of activation map.Defining a proper threshold to determine true activated regions, using common statistical processes, is achallenging task. This is due to a number of variation sources such as noise, artifacts, and physiologicalfluctuations in time series of fMRI data which affect spatial distribution of noise in an expected uniformactivated region. Spatial smoothing methods are frequently used as a preprocessing step to reduce theeffect of noise and artifacts. The smoothing may lead to a shift and enlargement of activation regions, and insome extend, unification of distinct regions.In this article, we propose a bootstrap resampling technique for analyzing single fMRI dataset with the aimof finding more accurate and reliable activated regions. This method can remove false positive voxels andpresent high localization accuracy in activation map without any spatial smoothing and statisticalthreshold setting.
bian).
l rights reserved.
ian MA, False positive control of activatedhing, Magn Reson Imaging (2013), http://dx
Functional magnetic resonance imaging (fMRI) non-invasivelyassesses signal changes in brain neuronal activities caused bychanges in local blood oxygenation levels. The purpose of designingan fMRI block paradigm is to let the blood hemodynamic response toreach its highest level and to remain in this situation until it falls byending the activation. In such a paradigm, the stimulus repeats manytimes during activation phase in order to keep the activation effectstable. Besides, rest block duration causes the hemodynamicresponse to reach its baseline. The time series of fMRI data acquiredduring the task presentation are expected to follow the activation-rest pattern, while they are affected by some unwanted within-subject variations.
Some statistical techniques such as t-test, f-test, and ANOVA havebeen introduced for finding the activated voxels following the task-related signal variations [1,2]. In fMRI data analysis, a statisticalthreshold determines a level to accept the activation regions amongthe whole triggered voxels. Due to the within-subject variations
caused by system noise, artifacts, head motion, and physiologicalfluctuations, finding accurate and reliable activated voxels from asingle fMRI dataset is difficult. Some of the false alarm voxels, causedby within-subject variation, can be eliminated by statistical groupingmethods used in higher level fMRI analysis.
While the statistical group analysis methods are proper tech-niques for reducing the within-subject variations, these methods arenot applicable for noise removing of the single fMRI data. Spatialsmoothing methods are used as a preprocessing step to reduce theeffect of noise and artifacts and improve the signal to noise ratio(SNR) in single fMRI analysis.
Although the spatial smoothing is one of the most importantsteps in preprocessing of the fMRI data, there are some drawbacks[3–5]. The size of the smoothing kernel should be carefully selected.The small filter is not effective enough to reduce the noise, while, abig smoothing kernel enlarge the activation region or it can mergetwo or more activation peaks when they are close to each other. Awide smoothing filter may lead to remove the small but importantactivations. The former can be problematic when the fMRI data areused in clinical diagnosis or surgical planning. The other disadvan-tage of smoothing is the shift of activation peaks [6]. Therefore, thesize of the spatial smoothing filter is crucial and should be carefully
voxels in single fMRI analysis using bootstrap.doi.org/10.1016/j.mri.2013.03.009
2 F. Darki, M.A. Oghabian / Magnetic Resonance Imaging xxx (2013) xxx–xxx
considered, especially, in clinical and neurosurgical investigations ofsingle fMRI data where the anatomical accuracy and a preciselocalization of fMRI result is vital.
Besides the spatial smoothing problem, the statistical threshold-ing of the activated voxels is critical. Small activated regionsmight beignored or the true activation size of an area might be reduced wheninappropriate threshold is applied. This also can have a vitalconsequence on clinical, neurosurgical and psychological environ-ments when the diagnosis should be applied on individual subject.
In spite of the unavoidability of the unwanted variations in fMRItime series, finding an accurate and reliable activation map from thefMRI data is very important. The reliability of fMRI data was assessedby test-retest analysis [7–10] in which the utilized task was repeatedseveral times and fMRI data were acquired using the same imagingparameters performed consecutively. Time-consuming scanning isthemain drawback of the test-retest methodwhichmakes it hard fora patient to bear. Moreover, the imaging parameters in each sessionshould be identical and the patient is expected to act similarly duringeach task repetition.
Resampling techniques called boosting or bagging [11,12] whichresample the data into many new created datasets have beenproposed to compute the reliability and confidence intervals of fMRIdata. These mimic the concept of the test-retest strategy plannedduring data acquisition [7–10] in the analysis stage. Resampling andreplacing the data, called jackknife and bootstrap [13], uses all theindependent samples to generate bootstrap datasets. One of themost important aims of the resampling techniques is to assess thereliability and reproducibility of a result. The jackknife method[11,12] was used to compute the reliability and confidence intervalsof fMRI parameters during bilateral finger tapping. Auffermann et al.[14] applied the bootstrap method to assess the significance of theself-organizing maps in clustering algorithm applied on event-related data. In another study, the bootstrap analysis was used toinvestigate the stability of the clusters found by K-means algorithmfor resting-state networks [15]. The consistency of componentsextracted from the Independent Component Analysis (ICA) methodwas also evaluated by this resampling technique [16]. In addition tothe application of the bootstrap method in assessing the reliabilityand stability of the fMRI results, this statistical technique has alsobeen used to generate a simulated dataset by mimicking theparameters of real fMRI data [17].
In this article, we aim to apply bootstrapmethod in order to find areliable and valid activation map from single fMRI data specificallywith the application of pre-surgical planning which requiresaccurate lateralization index as well as a precise localization ofactivation map. In single fMRI data analysis, the grouping methodsare not applicable for noise removal and besides the spatialsmoothing and threshold settings are challenging for finding theaccurate activated voxels. In this work, we generated many repeatedsamples from a single fMRI dataset before applying the analyzingprocess. This technique can be used as a noise removing method toavoid the false positive voxels and to find the accurate activationmap with no spatial smoothing or thresholding. The performance ofthis method was assessed using both simulated data and healthyhuman fMRI block design datasets.
2. Methods
2.1. Simulated fMRI data
Resting state fMRI data acquired from normal subject was used asa base 4D image for generating block design simulated fMRI datasets.Preselected regions on these 4D rest images were used as the initialdatasets for generating the block activated simulated datasets. We
Please cite this article as: Darki F, Oghabian MA, False positive conresampling in comparison to spatial smoothing, Magn Reson Imaging
assumed that, in these regions, there is no activation based on blockdesigned paradigm and the signal in these regions carries thephysiological fluctuations, head motion problem, realistic systemnoise and artifacts. The time series of the synthetic data weregenerated by convolving a known hemodynamic response function(HRF) model with a block signal consisting of 4 rests and 4activations, each containing 8 volumes. The simulated signals ofthe block activated paradigmwere then added voxel-by-voxel to thepreselected regions on the rest fMRI dataset, in order to mimic thenatural noise and artifacts in the simulated data. Eight differentsimulated data using different resting state images were generatedand used as “ground-truth” data for the evaluation of the bootstrapresampling as well as the comparison between the bootstrap andconventional methods.
2.2. Real fMRI data
Functional MR imaging were carried out using a 3 T Siemensscanner on 14 healthy subjects performing right and left handmovement in two separate imaging sessions. The subjects wereasked to perform motor block-design tasks during 64 sequentialvolumes consisted of 8 blocks (4 activation and 4 rest blocks), eachfor 24 s. The fMRI data were acquired by FOV = 24 × 21.7 cm2,slice-thickness = 3.6 mm and Matrix-size = 64 × 58. For anatom-ical imaging, T1-weighted spin-echo sequence was used togenerate high-resolution structural image with a pixel size of0.47 × 0.47 mm2, matrix size of 512 × 472, for 36 axial slices of3.6 mm thickness.
2.3. Bootstrap resampling method
In fMRI block designed paradigm, the stimulus repeats manytimes in activation blocks to keep the activation effect stable.Moreover, the rest block duration causes the hemodynamic responsereaches to its baseline. The time series of fMRI data acquired duringthe presentation of a block designed task are expected to follow therest/activation pattern. However, there are within-subject variationssuch as fluctuations and permutations in the time series data causedby system noise, artifacts, head motion, and low concentration ofsubject during task implementation, which affects activation pattern.Since the images acquired in each block (activation or rest) should beidentical to each other after removing their differences due to theinfluence of within-subject variations, they can be replaced by othersimilar images from other activation occasions. Therefore, boot-strapping is expected to decrease false positive voxels which do notshow up largely in various reproduced samples.
Here, bootstrap resampling method was used to generate a largenumber of datasets using the time series volumes of single fMRI datawithout any need to repeat the experiment multiple times. In thismethod, the volumes of each original fMRI dataset (both simulatedand real datasets) were divided into 4 sections consisting of rest,onset, activation, and fall-off parts called A, B, C and D, respectively(Fig. 1). The A, B, C and D sets was defined as below:
A ¼ ∪l
i¼1Ai ¼ A1;A2;…;Alf g
B ¼ ∪l
i¼1Bi ¼ B1;B2;…;Blf g
C ¼ ∪l
i¼1Ci ¼ C1;C2;…;Clf g
D ¼ ∪l
i¼1Di ¼ D1;D2;…;Dlf g
ð1Þ
where Ai, Bi, Ci and Di are the subsets of A, B, C and D and l is thenumber of their repetition. Considering 16 volumes for an active-restsequence, the number of each group was set to 4. The members of Ai,
trol of activated voxels in single fMRI analysis using bootstrap(2013), http://dx.doi.org/10.1016/j.mri.2013.03.009
Fig. 1. The volumes of the original fMRI data divided into 4 sections containing rest, onset, activation and fall-off parts.
3F. Darki, M.A. Oghabian / Magnetic Resonance Imaging xxx (2013) xxx–xxx
Bi, Ci and Di were randomly selected from A, B, C and D sets (i.e.,Ai ⊂ A, Bi ⊂ B, Ci ⊂ C, Di
⊂ D). Consequently, bootstrapped data-sets, S , were set as follows:
S′ ¼ A
′1;B
′1;C
′1;D
′1;…A
′l ;B
′l ;C
′l ;D
′l ;
n oð2Þ
These were then prepared for analyzing [18].
2.4. General linear model
The bootstrapped fMRI datasets were analyzed by General LinearModel (GLM) algorithm carried out using FEAT (fMRI ExpertAnalysis Tool), part of the FSL (FMRIB Software Library) software.The preprocessing methods including brain extraction from non-brain tissues using BET (Brain Extraction Tool), motion correction,and removing drifts from raw data were also performed. For thepurpose of comparing the results of bootstrap resamplingwith singleGLM analysis from the anatomical accuracy point of view, differentsmoothing kernel sizes equal to 0, 3 and 6 mm were utilized inconventional GLM analysis. While for bootstrapped data, spatialsmoothing was set to 0 to eliminate the influence of this process innoise removing of the bootstrap analysis.
2.5. Evaluation process
To evaluate the bootstrap resampling technique, both syntheticfMRI data and real BOLD datasets obtained from healthy subjectswere used. The bootstrap results without any spatial smoothnesswere compared with the corresponding results from a single GLManalysis using smoothing kernels equal to 0, 3 and 6 mm. Accuracy ofthe activated voxels were measured for both single and boot-strapped data.
The activation areas obtained by a significant p-value of 0.05were considered for all single and bootstrapped datasets. Athreshold, defined as the percentage of maximum Z-statistic value,was also set as a critical factor to discriminate between active andinactive voxels. In order to find the most reliable and valid activation
Fig. 2. Activation images from simulated data analyzed by single conventional GLM analysis
Please cite this article as: Darki F, Oghabian MA, False positive conresampling in comparison to spatial smoothing, Magn Reson Imaging
regions in the bootstrapped method, first, a number of repeatedareas were obtained using GLM analysis on the resampled timeseries data at a predefined threshold value. Subsequently, theactivated voxels, which were repeated in more than 95% of thecases, were considered as reliable.
The predefined regions from simulated activation areas as wellas the Talairach labeled regions of motor areas were used asreference masks for measuring accuracy of activated voxels.Accuracy was measured as a proportion of all true activated andtrue non-activated voxels (true positives and true negatives) to thewhole gray matter voxels.
In addition to activation accuracy, the percentage of signal changewas computed as a measure of activation enhancement. In this regard,the output image analyzed by single GLM analysis with no spatialsmoothing was used as the based image for each fMRI study to assessthe effect of bootstrapping on activation enhancement while it doesnot benefit from spatial smoothing. The percentage of activationenhancement was then calculated by subtracting the activation ofbased image from output image, divided by the based image.
3. Results
3.1. Simulated data
The results from analyzing simulated datasets for both single andbootstrap GLM methods are presented in Fig. 2 and Table 1. Thisfigure shows sample of activation images obtained by routine singleGLMmethod using spatial filtering with FWHMs of 0, 3 and 6 mm, aswell as the bootstrapped results without any spatial smoothing. Asshown, the spatial filtering used for suppression of noise enlarges thecluster size of activation regions. The table shows cluster sizes andpercentages of activation enhancement for 5 clusters obtained by theproposed methods. Although the spatial filtering can reduce noisyvoxels and increase the activation values, it further increases thenumber of false positives by widening the activation regions. Incontrast, the bootstrap resampling technique maintains the actualsizes of clusters as well as enhancing their activation values.
(using FWHMs = 0, 3, 6 mm) and bootstrap GLM analysis without spatial smoothing
trol of activated voxels in single fMRI analysis using bootstrap(2013), http://dx.doi.org/10.1016/j.mri.2013.03.009
4 F. Darki, M.A. Oghabian / Magnetic Resonance Imaging xxx (2013) xxx–xxx
Fig. 3 represents the accuracy of detecting activated voxels forsimulated data analyzed with a routine simple GLM method andbootstrap resampling strategy. As the graph shows, the bootstrapmethod provides an accuracy of over 93% even for a low thresholdlevel. This figure also shows the impact of bootstrapping in removingnoisy voxels compared to the effect of spatial filtering employed bythe single conventional GLM analysis. Bootstrappingmethod reducesthe false activated voxels caused by noise and artifacts, while it keepsthe activation size unchanged because of not using spatial smooth-ing. The measured accuracy for bootstrapped results was shown tobe also independent of the threshold set for discrimination of activevoxels from inactive ones (all above 90%). Considering the singleGLM method using various smoothing kernel size (FWHMs of 0, 3and 6 mm), Fig. 3 shows reduction in accuracy as the filter sizeincreases (from 95% to 80% at the threshold of 0.25 of max Z-statisticvalue). However, a good accuracy was achieved using bootstrappinganalysis with no need for spatial filtering.
3.2. Real fMRI assessment
In this experiment, binary masks of right and left motor areasobtained from Talairach atlas were used to compute the accuracy ofactive voxels. The same results as simulated data were obtained
ig. 3. Accuracy of the activated voxels obtained by single and bootstrap GLM analyses of simulated datasets using different activation thresholds. The predefined regions from
F simulated activation areas were used as reference masks for measuring accuracy of activated voxels.
Please cite this article as: Darki F, Oghabian MA, False positive control of activated voxels in single fMRI analysis using bootstrapresampling in comparison to spatial smoothing, Magn Reson Imaging (2013), http://dx.doi.org/10.1016/j.mri.2013.03.009
analyzing real fMRI data (Fig. 4). The accuracy of the bootstrapanalysis remains around 85% for both right and left hand move-ments. For the routine GLM analysis, the accuracy varies from 32% to75% for right hand, and from 34% to 84% for left hand movementdepending on thresholding level, when no smoothing was applied tothe original data. Accordingly, the bootstrap method showed to beindependent of both the smoothing process and the threshold level.
The image output from applying both techniques are shown inFig. 5. This figure clearly represents the power of bootstrapresampling analysis technique in eliminating false positive voxels,where no smoothing was applied. Moreover, as expected, thismethod is not significantly sensitive to the threshold. It is becausethe noisy voxels have already been removed during the bootstrapresampling procedure. In contrast, single conventional GLM analysisis shown to be very sensitive to thresholding; as a result, the numberof false positives reduces considerably by threshold variation from5% to 30% of its maximum Z-statistic value. The impact of spatialfiltering is also illustrated in these images in which higher filter sizeshows less scatter false voxels, and more integrated activation areas.
Consecutive slices of a sample subject activating right handmovement are presented in Fig. 6. A maximum threshold of 30%, asused in Fig. 5, was applied on the routine single GLM analysis. Thisfigure illustrate that bootstrapping technique can eliminate the noisyand non secure voxels. Although the conventional GLM analysis can
Fig. 4. Accuracy representation of real fMRI data analysis carried out by single conventional and bootstrap GLM analyses. Talairach labeled regions of motor areas were used asreference masks for measuring the anatomical accuracy of activated voxels.
5F. Darki, M.A. Oghabian / Magnetic Resonance Imaging xxx (2013) xxx–xxx
eliminate some of the false activated voxels by the use of spatialfiltering, this is in the expense of widening activation regions.Consequently, we believe that bootstrapping can generate morevalidated results by eliminating the less reliable voxels; still keepsthe activation size unchanged. In addition, the bootstrappingshowed to enhance the activation values compared to the single
Fig. 5. Thirty-third slice of one sample subject performing left hand movement, analyzed banalysis without spatial filtering, using different thresholds for Z-statistic values.
Please cite this article as: Darki F, Oghabian MA, False positive conresampling in comparison to spatial smoothing, Magn Reson Imaging
GLM analysis when using FWHM = 0. As shown in Table 2,bootstrap based activation values increased to 21.43% ± 1.47% and20.55% ± 1.03% for right hand movement and the left, receptively.
Grouping the fMRI results from hand movement was also used tocompare the two techniques after removing inter-subject variations.Comparing the results obtained from bootstrapping as illustrates in
y single GLM (with spatial filtering of FWHM = 0, 3, 6 mm) and bootstrap resampling
trol of activated voxels in single fMRI analysis using bootstrap(2013), http://dx.doi.org/10.1016/j.mri.2013.03.009
Fig. 6. Axial slices representing fMRI results obtained by right handmovement. The results of single GLM analysis with smoothing filter size of FWHM = 0, 3, 6 mm, and bootstrapresampling analysis without spatial filtering are presented.
6 F. Darki, M.A. Oghabian / Magnetic Resonance Imaging xxx (2013) xxx–xxx
Fig. 7 reveals that most of unreliable voxels can be corrected similarto the group analysis using fixed-effects.
4. Discussion
In this work, we performed a bootstrap resampling method onsimulated and real fMRI datasets to find reliable and reproducibleactivation maps without need of spatial smoothing. The methodwas shown also to be insensitive to statistical thresholding. Anaccuracy of 93% for detecting true active voxels in simulated datawas obtained for all the employed threshold levels. In contrast, theconventional GLM analysis showed accuracies from 54% to 95%depending on the level of threshold and the size of employedspatial filter. The reduction in accuracy is due to the enlargementof the activation regions (i.e. increase the number of falsepositives) occurring during spatial filtering. Besides, there is anincrease in accuracy for high activation thresholds due to increasein specificity (i.e. less false positive) of the detected voxels. So,high detection accuracy in single GLM analysis can only beobtained when an optimally low spatial filtering is used with anoptimum threshold value.
The proposed method showed to successfully remove the noisyvoxels and keep the size of activation region constant whileenhancing the activation values to about 12.5%. This can be justifieddue to the averaging nature of bootstrapping process on resampledtime series data. However, the smaller activation enhancementfound by the bootstrapping technique in comparison to the singledata analysis can be compromised by its size consistency. Moreover,
able 2ercentage of activation enhancement for fMRI data analyzed by single and bootstrapLM methods.
Method Activation enhancement in relation to the base image(single dataset analyzed with no spatial filtering)
Fig. 7. Thirty-first slice of a sample data analyzed by single, bootstrapped, and fixed-effect group analysis.
TPG
Please cite this article as: Darki F, Oghabian MA, False positive conresampling in comparison to spatial smoothing, Magn Reson Imaging
although the improvement of about 20% for activation of handmovements was obtained using the routine GLM analysis onspatially smoothed data, this was with the cost of activation regionenlargement.
Similar results were achieved for analyzing real fMRI dataobtained by motor area activation. In this regard, the bootstrapmethod attained accuracy above 85% for lowest employed thresholdand above 95% using an optimum threshold level.
Reliability and reproducibility of fMRI data has been assessedusing the test-retest method for motor [19], visual [20,21], language[22,23], memory, and cognitive tasks [24,25]. The main criterion forreliable assessment of these methods was based on the number ofsignificant voxels [26]. Reproducibility ratios of 20% [27], 54-59%[10], and 70% [28] have been reported for motor tasks. Alternatively,an activation reproducibility of 78% was reported for motor sensoryareas using four different paradigms [26]. Although fixed statisticalthresholds have been set for most of these experiments, a largevariation was obtained in reliability measurements of all theseexperiments.
The proposed bootstrap resampling method utilizes the samestrategy to provide a precise localization of activation regions but
trol of activated voxels in single fMRI analysis using bootstrap(2013), http://dx.doi.org/10.1016/j.mri.2013.03.009
7F. Darki, M.A. Oghabian / Magnetic Resonance Imaging xxx (2013) xxx–xxx
with no need to repeat the task and data acquisition several times.Since this resampling method eliminates the false alarm voxelsyielding the most reliable and reproducible voxels repeated at about95% in all bootstrapped samples, it can be used for single fMRI dataanalysis where the grouping methods are not applicable. Interest-ingly, the proposed method has potential to eliminate the within-subject variations comparable to the results obtained from fixed-effect group analysis.
The proposed method can be used routinely for single subjectdata analysis especially in pre-surgical functional evaluation whereno multi-session or multi-subject data exist. Based on our finding,we expect that both extension and location of the activation map bereasonably accurate and its size to be reliable due to lack of anyspatial filtering to remove noise and unwanted artifacts. Theenlargement and shift of activation areas affected by spatial filteringwas also shown in other studies [3–5]. Moreover, since the proposedmethod is quite independent of statistical thresholding, it can beexpected to eliminate any error occurred by threshold setting andstatistical implementation.
5. Conclusion
The bootstrap resampling GLM analysis method proposed in thispaper reduces false positive voxels and protects most reliable andreproducible voxels which appear to be active in more than 95% ofthe bootstrapped datasets. This method is quite insensitive tothreshold selection and attains a high accuracy in detection of activevoxels. In addition, the presented method boosts the activationvalues of activated regions, keeping their size and locationunchanged. Consequently, this method can be used in fMRI analysisof single subject for finding valid and reliable activation maps inorder to be used in neurosurgical and neuro-navigation planning.
References
[1] Ardekani BA, Kanno I. Statistical methods for detecting activated regions infunctional MRI of the brain. Magn Reson Imaging 1998;16(10):1217–25.
[2] Marchini JL, Ripley BD. A new statistical approach to detecting significantactivation in functional MRI. Neuroimage 2000;12(4):366–80.
[3] Fransson P, Merboldt KD, Petersson KM, Ingvar M, Frahm J. On the effects ofspatial filtering–a comparative fMRI study of episodic memory encoding at highand low resolution. Neuroimage 2002;16(4):977–84.
[4] Geissler A, Lanzenberger R, Barth M, Tahamtan AR, Milakara D, Gartus A, et al.Influence of fMRI smoothing procedures on replicability of fine scale motorlocalization. Neuroimage 2005;24(2):323–31.
[5] White T, O'Leary D, Magnotta V, Arndt S, Flaum M, Andreasen NC. Anatomic andfunctional variability: the effects of filter size in group fMRI data analysis.Neuroimage 2001;13(4):577–88.
Please cite this article as: Darki F, Oghabian MA, False positive conresampling in comparison to spatial smoothing, Magn Reson Imaging
[6] Mikl M, Mareček R, Hluštík P, Pavlicová M, Drastich A, Chlebus P, et al. Effects ofspatial smoothing on fMRI group inferences. Magn Reson Imaging 2008;26(4):490–503.
[7] Genovese CR, Noll DC, Eddy WF. Estimating test‐retest reliability in functionalMR imaging I: statistical methodology. Magn Reson Med 1997;38(3):497–507.
[8] Noll DC, Genovese CR, Nystrom LE, Vazquez AL, Forman SD, Eddy WF, et al.Estimating test‐retest reliability in functional MR imaging II: application tomotor and cognitive activation studies. Magn Reson Med 1997;38(3):508–17.
[9] Rombouts S, Barkhof F, Hoogenraad F, Sprenger M, Valk J, Scheltens P. Test-retestanalysis with functional MR of the activated area in the human visual cortex.American journal of neuroradiology 1997;18(7):1317–22.
[10] Yetkin FZ, McAuliffe TL, Cox R, Haughton VM. Test-retest precision of functionalMR in sensory and motor task activation. American journal of neuroradiology1996;17(1):95–8.
[11] Bappal A, Biswal BB. Modified bootstrap resampling technique consideringtemporal correlation in fMRI. Proc Intl Soc Mag Reson Med 2004;11:2532.
[12] Biswal BB, Taylor PA, Ulmer JL. Use of jackknife resampling techniques toestimate the confidence intervals of fMRI parameters. J Comput Assist Tomogr2001;25(1):113.
[13] Efron B, Tibshirani RJ. An Introduction to the Bootstrap. Chapman & Hall/CRCmonographs on statistics & applied probability; 1994.
[14] Auffermann WF, Ngan SC, Hu X. Cluster significance testing using the bootstrap.Neuroimage 2002;17(2):583–91.
[16] Ylipaavalniemi J, Vigário R. Analyzing consistency of independent components:an fMRI illustration. Neuroimage 2008;39(1):169–80.
[17] Bellec P, Perlbarg V, Evans AC. Bootstrap generation and evaluation of an fMRIsimulation database. Magn Reson Imaging 2009;27(10):1382–96.
[18] Darki F, Oghabian M. Accurate activation map detection using bootstrapresampling of single fMRI data. Engineering in Medicine and Biology Society2009.
[19] Loubinoux I, Carel C, Alary F, Boulanouar K, Viallard G, Manelfe C, et al. Within-session and between-session reproducibility of cerebral sensorimotor activa-tion: a test–retest effect evidenced with functional magnetic resonance imaging.J Cereb Blood Flow Metab 2001;21(5):592–607.
[20] Moser E, Teichtmeister C, Diemling M. Reproducibility and postprocessing ofgradient-echo functional MRI to improve localization of brain activity in thehuman visual cortex. Magn Reson Imaging 1996;14(6):567–79.
[21] Specht K, Willmes K, Shah NJ, Jäncke L. Assessment of reliability in functionalimaging studies. J Magn Reson Imaging 2003;17(4):463–71.
[22] Rutten G, Ramsey N, Van Rijen P, Van Veelen C. Reproducibility of fMRI-determined language lateralization in individual subjects. Brain Lang2002;80(3):421–37.
[23] Veltman D, Friston K, Sanders G, Price C. Regionally specific sensitivitydifferences in fMRI and PET: where do they come from? Neuroimage2000;11(6):575–88.
[24] Kiehl KA, Liddle PF. Reproducibility of the hemodynamic response to auditoryoddball stimuli: a six‐week test–retest study. Hum Brain Mapp 2003;18(1):42–52.
[25] MachielsenW, Rombouts SARB, Barkhof F, Scheltens P, Witter MP. FMRI of visualencoding: reproducibility of activation. Hum Brain Mapp 2000;9(3):156–64.
[26] Havel P, Braun B, Rau S, Tonn JC, Fesl G, Brückmann H, et al. Reproducibility ofactivation in four motor paradigms. J Neurol 2006;253(4):471–6.
[27] Ramsey NF, Tallent K, van Gelderen P, Frank JA, Moonen CTW, Weinberger DR.Reproducibility of human 3D fMRI brain maps acquired during a motor task.Hum Brain Mapp 1996;4(2):113–21.
[28] Mattay VS, Frank JA, Santha A, Pekar JJ, Duyn JH, McLaughlin AC, et al. Whole-brain functional mapping with isotropic MR imaging. Radiology 1996;201(2):399–404.
trol of activated voxels in single fMRI analysis using bootstrap(2013), http://dx.doi.org/10.1016/j.mri.2013.03.009
