...design teams should anticipate learners’ interest, motivation and engagement...the best visualizations...are fluid, dynamic artifacts that respond to the need for a different view or for more detailed information...signs - things (real and/or imaginary) extend some sensory, physical, or intellectual capacity...we do not perceive meaning-bearing differences in absolute ways, but rather in relational terms…[signs] are semiotically constrained by culture...we are in a constant process of deciphering what something ‘stands for’...forward thinking research considers human and the computer together as a problem solving system...we are experiencing the miniaturization of the world through media...

Cobley, 2010; Sawyer, 2014; Ware, 2014 1

Semiotics, Information and Interactive Design: Informing Geoscience Education, Multimedia Teaching and Learning

John C. BedwardBuena Vista University

Assistant Professor STEM EducationChair Mathematics, Science and STEM

EARTH EDUCATORS’ RENDEZVOUS 2016, UNIVERSITY OF WISCONSIN, MADISON 2

Contents

● Sourcing Science Content● Learning with Rich Media● Design Science● Getting Started

3

Source, Disciplinary Core Ideas

● Next Generation Science Standards - 3D Learning

● Natural and social/anthropocentric phenomenon (Grand Challenges)

● Earth Science Literacy Documents

Achieve, 2016; NAGT, 2016; NCSTATE, 2014; NAP, 2016; U.N. 2015, Wikipedia, 2016

A natural arch produced by erosion of differentially weathered rock in Jebel Kharaz (Jordan)

THE NATIONAL ACADEMIES PRESS

Assessment Instruments

4

Abridged Geoscience Video Assessment Rubric

● Concept significance: The concept is significant, relevant, worthwhile to students and developmentally appropriate for learning essential concepts and skills…

● Scientific accuracy: The underlying scientific concept is clear, accurate, error free, unambiguous…

● Alignment of objectives, activity and assessment:

● Robustness: The video is easy to use, across multiple devices; the activity includes all components (e.g., video, worksheets, assessments)...

● Activity description: Sufficient content information is provided allowing the instructor to make an informed decision about whether or not this is an appropriate activity to use in their learning situation.

Content

5

Goal of Multimedia Learning

● Building coherent mental structures● The construction of schemata (formation and/or embellishment of new

schemata)● Automaticity of schemata

Mayer, 2014 6

Cognitive Theory of Multimedia Learning

Mayer, 2014

Dual Channel: separate channels for processing auditory and visual information Limited Capacity: limited in the amount of information each channel can process Active Processing: attend to relevant incoming information, organize information into coherent mental structures and integrate mental representations with other information 7

Neuro -Learning Science Education

Learning pathways...

● Sensory - Affective - Reward - Attention & Memory - Language & Mathematics - Executive Function - Social○ Executive Function (higher-order thinking functions)

■ Analysis and synthesis■ Metacognition■ Mental flexibility-shifting strategies and adjusting to change■ Working memory■ Handling ambiguity and uncertainty

Zadina, 2014; Sawyer, 2014 8

The Visual Field

Attention

● “...humans do not perceive much unless they have at least some expectation and need to see.”

● Attention to objects, expands and contracts based on task, information in display and level of stress in the observer

● Useful field of view (UFOV) - fixate on detail and glance for patterns● Cognitive load can exacerbate tunnel vision● Motion is helpful in attracting attention (i.e., animation)

Google, 2016; Sawyer, 2014; Ware, 2004 9

Sound, Hearing, Learning

Auditory Scene Analysis

● Build separate mental representations of the events from the sound mixture

● Infer backwards from the sounds to the event● Integrate mental representations of auditory

scene w/ accompanying visual scene (i.e., auditory grouping and auditory schema)

Coren, Ward & Enns, 2004 10

Gestalt, Semiotics and Multimedia Design

How we see patterns, reason with signs to design student learning experiences

● Proximity - Similarity - Connectedness - Continuity - Symmetry - Closure - Relative Size - Figure and Ground

● Present data easy to perceive, users take advantage of pattern finding○ Visual query patterns

● Minimize cognitive impact of interface, users focus on problem solving○ Cost of navigation (i.e., hypertext links, mouseovers, walking and flyovers)

● Optimize interface for rapid information seeking○ Constellations (i.e., interactive technique to access large amount of semantic information)

Cobley, 2010; Mayer, 2014; Sawyer, 2014; Ware, 2004 11

● Design teams, anticipate differences in learners’ interest, motivation and engagement, and include problem features that are effective for each

○ Make clear the contents utility, its relevance, and/or by triggering their interest through novelty, challenge, surprise, complexity, or uncertainty

○ Provide scaffolding that directly engages the student with the content and supports them to stretch their understanding

Cobley, 2010; Mayer, 2014; Sawyer, 2014; Ware, 2004 12

Abridged Geoscience Video Assessment Rubric

● Syncing visuals and narration: Narration and pictures are displayed to encourage understanding and minimize misunderstanding.

● Use of multiple modes of communication:

● Minimized redundancy: Removed all unnecessary visuals, text, narration and/or special effects.

● Scene layout, sequence & transition: Information is consistently displayed, organized and communicated using a combination of color, font style, graphic elements, special effects and whitespace.

● Lighting & image quality: Brightness, darkness, contrast and resolution were corrected for all images and scenes.

Design

13

Getting Started...

14

Pre-production

Background ResearchAsset Collection

Script/Storyboard

Production

Recording

Post-production

EffectsEdit

Publish

References

● Achieve (2016). The Next Generation Science Standards. Retrieved July 2016 from http://www.nextgenscience.org/.● Cobley, P. (2010). The Routledge Companion to Semiotics. New York: Routledge, Taylor & Francis Group.● Coren, S., Ward, L. & Enns, J. (2004). Sensation and Perception (6th Edition). USA: John Wiley & Sons, Inc.● Mayer, R. (2014). The Cambridge Handbook of Multimedia Learning (2nd Edition). New York, NY: Cambridge University

Press.● National Academies of Science, Engineering and Medicine (The) (2016). National Academy of Engineering Grand

Challenges. Retrieved July 2016 from http://www.engineeringchallenges.org/challenges.aspx.● North Carolina State University (2016). Assessment Instrument Information Page. Retrieved July 2016 from https://www.

ncsu.edu/per/TestInfo.html.● Sawyer, K. R. (2014). The Cambridge Handbook of the Learning Sciences (2nd Edition). New York, NY: Cambridge

University Press.● United Nations (2016). The United Nations. Retrieved July 2016 from https://sustainabledevelopment.un.org/sdgs.● Ware, C. (2004). Information Visualization: Perception for Design (2nd Edition). San Francisco: Morgan Kaufman.● Wikipedia (2016). Wikipedia: The Free Encyclopedia. Retrieved July 2016 from https://en.wikipedia.org/wiki/Weathering.● Zadina, J. N. (2014). Multiple Pathways to the Student Brain: Energizing and Enhancing Instruction. USA: Jossey Bass.

15