Visual Communication: theoretical issues
Research Seminar Master Informatiekunde13/09/06
Leonie Bosveld-de SmetHumanities Computing, RuG
04/22/23 Visual Communication 2
Question Pictures can be worth ten thousand words.
Visual representations can be superior to verbal ones in thinking (problem solving), teaching, and communication.
Can benefits observed in other areas (esp. science) be exploited in other fields?
04/22/23 Visual Communication 3
Outline talk
Characteristics of graphical versus sentential languages
Facilitating effects of visual representations
Visual communication: a semantic view
Conclusion and discussion
04/22/23 Visual Communication 4
Theoretical concepts Key words:
Representation system Interpretation function Form of representation Perception of representation Medium modality
Piece of information
Structured information
Representation system
Form of representationlink
04/22/23 Visual Communication 5
Media and modality contrasts Stenning and Inder(1995) “Medium” how is a representation perceived?
“Modality” how is a representation interpreted?
04/22/23 Visual Communication 6
Graphical vs Linguistic Representations
Stenning and Inder (1995) the more constrained, the less expressive, the more
tractable
Stenning and Oberlander (1995) limited abstraction, aided “processibility”
Shimojima (2004) Free ride properties Auto-consistency Specificity Meaning derivation properties
04/22/23 Visual Communication 7
Free Ride Property
Defeated(Jon,Bob) & Lost_to(Ken,Bob)FOL
Jon defeated Bob and Ken lost to Bob.English
Jon Bob KenPD
Express information:Jon defeated Bob.Ken lost to Bob.
04/22/23 Visual Communication 8
SpecificityExpress information:Jon defeated Bob.Ken defeated Bob.
Defeated(Jon,Bob) & Defeated(Ken,Bob)FOL
Jon defeated Bob and Ken defeated Bob.English
Jon Ken BobPD Ken Jon Bob? ?
04/22/23 Visual Communication 9
Meaning Derivation PropertyExpress information:Jon defeated Bob Bob defeated KenGil defeated Jon Ken defeated Ron
Defeated(Jon,Bob) & Defeated(Bob,Ken) & Defeated(Gil,Jon) & Defeated(Ken,Ron)FOL
Jon defeated Bob and Bob defeated Ken and Gil defeated Jon and Ken defeated Ron.English
Gil Jon Bob Ken RonPD
04/22/23 Visual Communication 10
Benefits of graphical systems Automaticity of inference Ease of consistency inferences Difficulty to express “abstract” information Richness of semantic content
“Transparent” problem solving Computational offloading Amplification of cognition
04/22/23 Visual Communication 11
Monk Puzzle (Winn, 1987)“A monk went to the temple at the top of a holy mountain to meditate and pray. He started out early one morning along the path that led up to the temple. Because he was an old man, and the way was steep and arduous, he frequently slowed his pace, and even sat and rested a while beside the path. Toward evening, he came to the temple at the top.After several days of meditation and prayer, it was time for him to leave. Early in the morning, he set off back down the path. Again, he frequently changed his pace and rested by the way. He arrived back at the bottom in the evening.Show that there is one single point on the path up the mountain where the monk will be at precisely the same time both when he goes up and when he comes down.”
04/22/23 Visual Communication 12
“See” the solution
04/22/23 Visual Communication 13
Larkin and Simon (1987) “Why a Diagram is (Sometimes) Worth Ten Thousand
Words”
Diagrammatic vs. Sentential representations Informational equivalence, computational offloading
Pulley problem Geometry problem
Grouping of information Exploitation of space Easy perceptual inferences
04/22/23 Visual Communication 14
Using vision to think Kosslyn (1994)
Scatter plots...employ point symbols (such as dots, small triangles, or squares) as content elements. The height of each point symbol indicates an amount. These displays typically include so many points that they form a cloud; information is conveyed by the shape and the density of the cloud
Simultaneous presentation of local information and global information implied by the local information.
04/22/23 Visual Communication 15
Visual Communication
Raw Data
Data Tables
Visual Structures
Views
Data Transformations
Visual Mappings
View Transformations
04/22/23 Visual Communication 16
Semantics of pictures:Wang’s view
Graphical Domain Application Domain
12
3
1: geometrical characterization of pictures
2: interpretations: graphical entities – application domain objects
3: specifications: application domain concepts – picture classes
04/22/23 Visual Communication 17
Natural vs. Unnatural Link
a
b
c
ab
c
04/22/23 Visual Communication 18
Misleading Link
Bill
Charles
Gregory
04/22/23 Visual Communication 19
Kinds of visual communication Via interpretations:
Illustration Demonstration Reasoning
Via picture specifications: Geometric constraint maintenance Design Computational art
04/22/23 Visual Communication 20
RequirementsMeaningful and Easily Tractable diagrams
Clear delimitation of critical information
Description of information by well-defined notions and properties (elements of application domain)
Graphical devices suitable for representation of information (elements of graphical domain)
Explicit, natural, not misleading link between graphical domain and application domain
04/22/23 Visual Communication 21
Graphical domain Research in information visualization and diagrams and
diagrammatic reasoning
Bertin Tufte Shneiderman
Blackwell Hegarty Scaife & Rogers Novick
04/22/23 Visual Communication 22
Possible application areas Language instruction
Linguistic research
History instruction
Historical research
04/22/23 Visual Communication 23
Towards an automated system for diagram research
“When we saw the spaceship, we stopped the car.”
e1: see
e2: stop
t t’ now
e2: ring
s: making lunch
e1: make lunch
t t’ now
t now
“Two men delivered the sofa. I had already paid for it.”
“David was making lunch when the phone rang.”
e1: deliver
e2: pay s: have paid
04/22/23 Visual Communication 24
3-D Syntactic trees