Studying Relationships Between Human Gaze, Description, and Computer Vision
Kiwon Yun1, Yifan Peng1
Dimitris Samaras1, Gregory J. Zelinsky1,2, Tamara L. Berg3 1Department of Computer Science, Stony Brook University
2Department of Psychology, Stony Brook University3Department of Computer Science, University of North Carolina
Computer Vision and Pattern Recognition (CVPR) 2013
OverviewUser behavior while freely viewing images contains an abundance
of information - about user intent and depicted scene content.
• Human gaze “where” the important things are in an image.
• Description “what” is in an image, which parts of an image are important to the viewer.
• Computer vision “what” might be “where” in an image. However, it will always be noisy and have no knowledge of importance.
OverviewUser behavior while freely viewing images contains an abundance
of information - about user intent and depicted scene content.2. From these exploratory analyses,
we build prototype applications for gaze-enabled object detection and annotation.
Human gaze
DescriptionAn old black woman wearing a turban and a headdress and a dog next to her wearing the same red headdress
Image content
1. We conduct several experiments to better understand the relationship between gaze, description, and image content.
SUN09• 104 images, free-viewing for 5 seconds• eye movements from 8 observers, each of whom provided a scene description.
Datasets
PASCAL VOC• 1,000 images, free-viewing for 3 seconds• eye movements from 3 observers• 5 natural language descriptions per image from different observers.
Experiments and Analyses
• People are more likely to look at people, other animals, televisions, and vehicles.• People are less likely to look at chairs, bottles, potted plants, drawers, and rugs.• Animate objects are much more likely to be fixated than inanimate objects (0.636 vs. 0.495).
Gaze vs. Object Type
Probability of being fixated when present for various object categories (top: PASCAL, bottom: SUN09)
Experiments and Analyses
Gaze vs. Location on Objects
person horse bird bus train tv
bicycle chair table cabinet curtain plant
• Animate objects are much more likely to be described than inanimate objects (0.843 vs. 0.545).
What objects do people describe?
Experiments and Analyses
What is the relationship between gaze and description?
P (fixated | described) P (described | fixated)PASCAL 0.866 0.952SUN09 0.737 0.725
S1: A man is reading the label on a beverage bottle. S2: A man looking at the bottle of beer that he is holding. S3: The man in a white tee shirt is holding a beer bottle and looking at it. S4: The scraggly haired man is holding up and admiring his bottle of beer. S5: Young man with curly black hair holding a beer bottle.
Fixated objects: bottle, person. Described objects: bottle, person
Gaze-Enabled Computer Vision
Analysis of Human Gaze with Object Detectors• Potential for gaze to increase the performance of object detectors varies by
object category.
Gaze-Enabled Computer Vision
Gaze-Enabled Object Detection and Annotation• Combine gaze and automated object detection methods to create a collaborative
system for detection and annotation.
Gaze-Enabled Computer Vision
Gaze-Enabled Object Detection and Annotation
Conclusion
• Through a series of behavioral studies and experimental evaluations, we explored the information contained in eye movements and description, and analyzed their relationship with image content.
• We also examined the complex relationships between human gaze and outputs of current visual detection methods.
• In future work, we will build on this work in the development of more intelligent human-computer interactive systems for image understanding.
[1] Studying Relationships Between Human Gaze, Description, and Computer Vision, Kiwon Yun, Yifan Peng, Dimitris Samaras, Gregory J. Zelinsky, and Tamara L. Berg, Computer Vision and Pattern Recognition (CVPR) 2013 (Oregon/USA)
[2] Specifying the Relationships Between Objects, Gaze, and Descriptions for Scene Understanding, Kiwon Yun, Yifan Peng, Hossein Adeli, Tamara L. Berg, Dimitris Samaras, and Gregory J. Zelinsky, Visual Science Society (VSS) 2013 (Florida/USA)