Understanding Spatial Correlation in Eye-fixation
maps for Visual Attention in Videos
Tariq Alshawi*, Zhiling Long, and Ghassan AlRegib
Multimedia and Sensors Lab (MSL)
Center for Signal and Information Processing (CSIP)
School of Electrical and Computer Engineering
Georgia Institute of Technology
Outline
1. Introduction to Human Visual Attention• Motivation
• Applications
2. Data• Dataset
• Eye-fixations Maps
3. Spatial Correlation• Modeling
• Results and discussion
4. Conclusions
2
Introduction to Human Visual Attention:
Motivation
3
(Diagram from http://ivrgwww.epfl.ch/supplementary_material/RK_CVPR09)
Introduction to Human Visual Attention:
Applications
4
Auto-Cropping2Compression1
1. Chenlei Guo; Liming Zhang, "A Novel Multiresolution Spatiotemporal Saliency Detection Model and Its Applications in Image
and Video Compression," in Image Processing, IEEE Transactions on , vol.19, no.1, pp.185-198, Jan. 2010
2. F. W. M. Stentiford, “Attention based Auto Image Cropping,” Workshop on Computational Attention and Applications, ICVS,
Bielefeld, March 21-24, 2007.
Introduction to Human Visual Attention:
Uncertainty Framework
5
T. Alshawi, Z. Long, and G. AlRegib, "Unsupervised Uncertainty Analysis For Video Saliency Detection" the 49th Asilomar Conference
on Signals, Systems and Computers, Pacific Grove, CA, Nov. 8-11, 2015
Dataset
• CRCNS
• 50 video clips, 5-90 seconds
• Street scenes, TV sports, TV
news, TV talks, video games,
etc.
• Ground truth by human subjects
(eye tracking)
6
Preparing Eye-fixation maps
7
[#] [x,y] [t,N]
[#] [x,y] [t,N]
[#] [x,y] [t,N]
[#] [x,y] [t,N]
[#] [x,y] [t,N]
[#] [x,y] [t,N]
[#] [x,y] [t,N]
Eye-Fixation Data
240 Hz
Spatial Correlation:
Spatiotemporal neighbors
8
Frame# k Frame# k+1Frame# k–1
Pixel of
Interest
Spatial Correlation:
Modeling
9
Frame# k Frame# k+1Frame# k–1
Pixel of
Interest
Frame# k Frame# k+1Frame# k–1
Pixel of
Interest
Temporal NeighborsSpatial Neighbors
Spatial Correlation:
Results (Spatial)
10
gamecube_07
Spatial Correlation:
Results (Spatial)
11
sccadetest_01
Spatial Correlation:
Results (Spatial)
12
tv-news_03
Spatial Correlation:
Results (Temporal)
13
Conclusions
• Insights into visual attention mechanisms for videos can help improve saliency-dependent video processing applications
• Analysis of eye-fixation maps correlation, independent of video content
• Experiments show substantial correlation between saliency of a pixel and that of its direct neighbors
• Eye-fixation map correlation is significantly affected by the video’s content and complexity
• Eye-fixation correlation can be used as a measure of the reliability of detected saliency, thus, optimize saliency-based video processing applications
14
Questions?
15