Exploring EEG for Object Detection and Retrieval
Eva Mohedano Amaia Salvador Sergi Porta Xavi Giró Graham O’Healy Kevin McGuinness Noel O’Connor Alan Smeaton
ACM International Conference on Multimedia Retrieval (ICMR) 2015.June 23-26, 2015. Shanghai, China.
Outline
● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work
2
Outline
● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work
5
Related Work: Rapid Image RetrievalWang, J., Pohlmeyer, E., Hanna, B., Jiang, Y. G., Sajda, P., & Chang, S. F. (2009, October). Brain state decoding for rapid image retrieval. In Proceedings of the 17th ACM international conference on Multimedia (pp. 945-954). ACM.
6
Related Work: Rapid Image RetrievalJun Yang, “A General Framework for Classifier Adaptation and its Applications to Multimedia”. Phd thesis. Carnegie Mellon University (2009).
Section 7.2: Adaptation of EEG-based Relevance ModelsCross-user adaptation to avoid re-training EEG-based SVM classifiers for relevance prediction.
7
Related Work: EEG Object Detection
Ref: Kapoor, Ashish, Pradeep Shenoy, and Desney Tan. "Combining brain computer interfaces with vision for object categorization." Computer Vision and Pattern Recognition, 2008. CVPR 2008. 9
Outline
● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work
10
Methodology: DatasetSubset of TRECVID INS 2013 Dataset● 3 Instances (4 local regions/instance)● For each topic, 1000 Images
○ 50 relevant (targets) and 950 non-relevant (distractors)
11
Methodology: Image Sorting
Unknown
200=10/190 images @ 5 Hz = 40 s200=10/190 images @ 10 Hz = 20 s
...Rest...
Round
Time between relevant (targets)
10/190 guarantees 5% of target, but they may not be apart.
...Rest Rest... Rest... ...
5 rounds x 200 = 1,000 images @ 5 Hz -> 200 seconds 13
Methodology: Feature Vectors
Cut EEG activity related to visual event
From 200 ms to 1 second after the target presentation. 14
Methodology: Feature Vectors
1
2
3
32
... ...
Band-pass 0.1-20Hz
Downsample1000 → 250 Hz
Downsample250 → 20Hz
Channel concatenation512D vector
15
Outline
● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work
17
Object Detection: Early results
Images with high classifier scores (yet not relevant) for query 1.19
Object Detection: User diversityReceiver Operating Characteric prove EEGs are valid, but with significant variations among users.
20
Outline
● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work
23
Retrieval within our dataset
Linear SVM
EEG: Sorting according to SVM confidence score
Mouse: Top: Clicked images (relevant)Bottom: Observed but not clicked imagesMiddle: Remaining images randomly sorted
25
Retrieval in a larger dataset
28
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105).Software: Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., ... & Darrell, T. (2014, November). Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia(pp. 675-678). ACM. [web]
Retrieval in a larger datasetEEG
Top 10: Positives
Bottom 100: Negatives
Clicked: Positives
Observed & unclicked:Negatives
Linear SVM...
Testing:
29
Outline
● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work
32
Outline
● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work
37