November 6, 2017 Sam Siewert

CS415Human Computer Interaction

Lecture 11 – Advanced HCIIntro to Cognitive Models and More Human Interaction from Computers

Minute Paper - Cyborg Question?1. … “useful form of HCI” … “can help people with disabilities” … “for

example, robotic prosthetics for veterans”2. … “sounds interesting and beneficial” … but … “will change how we do

things” … “how much dependency are we willing to put into” … “we can live without our phones” … “The same may not be true with cybernetics. We may never go back” … “may have some unforeseen consequences”

3. “definitely useful in” … “current state” … “not sure how useful it would be for HCI” … e.g. “Heads Up Display”…, but … “risk” … “could be security” … “Could it be hijacked?”

4. “when used to restore” … “beneficial”, … but … “used to enhance or ‘just because’ they are not always beneficial” and if “we develop reliance, that’s when these technologies lose their positive qualities”

5. “I don’t see why it would be a bad thing” … “subject(ed) to the same slow social acceptance” … “cybernetics will probably follow the trend of phones as they become smaller and more powerful”

6. “a general positive” …”used to enhance the human experience” …”They must be heavily tested” … “limited only by imagination and technological bounds”

7. “extremely useful”… but … “can go too far” …”especially for body modification” and “should not be used in a way that takes away quality of life”

Sam Siewert 2

AssignmentsAssignment #5 – Propose Group Project (Groups of 2, 3 or 4)– Keep it Simple, Provide Key HCI Design Features you

Hypothesize to be more Usable (Better) than Some Existing HCI or Try Something New (advanced HCI)

– Must Design a Usability Experiment [Positive or Negative Outcome OK, focus is design, prototyping to evaluate]

Exam #2 after Thanksgiving [Chapters covered since Exam #1]

Assignment #6 – Project

Final Oral Exam – Presentation of Project

Sam Siewert 3

This Week …

Chapter 12 – Cognitive Models– GOMS – Goal, Operators, Methods and Selection

Goal - describing what the user wants to achieveOperators - basic actions that the user must perform in order to use the systemMethods - several ways in which a goal can be split into sub-goalsSelection – choice of method and prediction of which will be used

Sam Siewert 4

Poor Software-Engineering PracticesHCI is an Aspect of Software EngineeringHMI is and Aspect of Systems EngineeringSoftware and Hardware Must be Co-DesignedSoftware CMVC and SQA Should be RigorousNo Software Guru Should be the Only One Who Knows How the Code “Works”

Sam Siewert 5

Assignment #5 - ObjectivesExplore Root-causes of Therac-25 Accidents

Propose a Final Team Exercise1. provide a design and prototype that suggests how to improve a

well-known existing HCI 2. provide a design and prototype to address a current

(contemporary) HCI for an application such as intelligent-transportation, UAV/UAS operations, wearable computing (e.g. smart-watch), etc.

3. design and experiment to evaluate whether a contemporary or proposed HCI design and implementation does in fact provide advantage over previous alternatives (ERAU IRB guidelines)

Sam Siewert 6

Cognitive Models - IntroductionHierarchical Models – Task and Goal Structure

Linguistic Models – System Grammar

Physical and Device Models – Human Motor Skills

Cognitive Models – Combine All 3 of the Above

Sam Siewert 7

1) GOMS = Goals, Operators, Methods, Selection

Sam Siewert 8

GOMS - Selection

Sam Siewert 9

GOMS - EvaluationNew HCI Design Efficiency

Sam Siewert 10

2) Linguistic ModelsBNF (Backus Naur Form) – Syntax

E.g.– 0123456789– 0123456789.0123456789– 0123456789.0123456789E+0123456789

Sam Siewert 11

Linguistic Models - HCIInteractive Specification of a US Postal AddressForm Text Entry, Pull-downs, etc.

Sam Siewert 12

https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form

Change of Address Example

Sam Siewert 13

https://www.usps.com/manage/forward.htm

Linguistic SystemsSyntax – Rules for Writing ExpressionsSemantics – Expression Meaning Chomsky’s Hierarchy

Sam Siewert 14

In Practice, Limited to Regular Expressions and CFG

Natural Languages Use both CFG and CSG – Noam Chomsky [Linguistics, Strong Influence on Computer Science and Programming Languages, Activist]Alonzo Church – Lambda CalculusKurt Gödel – Mathematics Related to Languages and Formal Systems -CompletenessAlan Turing – Turing Machine and Computability

Sam Siewert 15

http://en.wikipedia.org/wiki/Chomsky_hierarchy

Formal Grammars

3) Physical and Device ModelsKeystroke Level Model

1. Finding and Striking Keys (Combinations Ctrl+Alt+Del)2. Mouse Button3. Moving Cursor to a Target (Fitts’ Law)4. Mouse to Keyboard Hand Transition5. Tracing with Mouse6. Thought Prior to Action7. System Response Latency

Sam Siewert 16

Physical EfficiencyGOMS Task – Time to Complete

Sam Siewert 17

Example AnalysisKeystroke Short-cut vs. Point and Click

Sam Siewert 18

Interactive Cognitive Systems

Sam Siewert 19

Philip Barnard– ICS provides a model of perception, cognition and action– not intended to describe the user in terms of action sequences– ICS provides a more holistic view of the user as an information-

processing machine– emphasis on how easy particular procedures of action

sequences become– as they are made more automatic within the user

Interdisciplinary Study and Research– Computer Science – Automation, Programming Languages, AI,

HCI, Systems, Machine Learning– Cognitive Psychology – Human Perception, Thought, Reason– Brain Physiology – fMRI, functional Magnetic Resonance

Imaging– Interesting Recent Projects – EPFL Blue Brain, IBM Watson

Reality – Hybrid of 3 Models and …HCI – GOMS + Linguistic + Physical + IntelligenceNewell and Simon [CMU, AI]Power of Recursion, 1958 NSS Chess – E.g. Minimax Search

Sam Siewert 20

Herbert Alexander Simon, (June 15, 1916 – February 9, 2001)was an American scientist and artificial intelligence pioneer, economist and psychologist. Professor, most notably, at the Carnegie Mellon University, Pittsburgh, Pennsylvania, which became an important center of AI and computer chess, ...Herbert Simon received many top-level honors, most notably the Turing Award (with Allen Newell) (1975) for his AI-contributions [1] and the Nobel Memorial Prize in Economics for his pioneering research into the decision-making process within economic organizations (1978) [2

Allen Newell, (March 19, 1927 - July 19, 1992)was a American researcher in computer science and pioneer in the field of artificial intelligence and chess software [1] at the Carnegie Mellon University, Pittsburgh, Pennsylvania. In 1958, Allen Newell, Cliff Shaw, and Herbert Simon developed the chess program NSS [2]. It was written in a high-level language. Allen Newell and Herbert Simon were co-inventors of the alpha-beta algorithm, which was independently approximated or invented by John McCarthy, Arthur Samuel and Alexander Brudno [3].

https://chessprogramming.wikispaces.com/Herbert+Simon

About 40 Years Later …

From Algorithm to Computer Oponnent– YouTube Documentary– Randy Moulic’s Team IBM / CMU

– Kasparov Requested a Re-match, No Response from IBM– IBM’s Page - https://www.research.ibm.com/deepblue/– Controversy – Game 2, Wired on “bug”, Conditions– Chess today – http://en.playchess.com/ (GNU Chess – 1989-92)

Sam Siewert 21

Is Deep Blue Intelligent?

22

“I used to think chess required thought. Now, I realize it doesn’t.”Douglas Hofstadter, 1997

“… (Deep Blue has) a weird kind (of intelligence) …”Garry Kasparov, 1996

For more information about Deep Blue, please look at : “Deep Blue”, Murray Campbell, Joe Hoane, Feng-Hsuing Hsu, Artificial Intelligence, 134, (2002), p57-83.

“Does it really matter if unbounded brute force analysis can achieve similar results? “ Randy Moulic

From Dr. Randy Moulic’s Slides – Deep Blue, Seoul Korean, June 2014

Advanced HCI … Future

Microsoft Research HCI Vision

Interesting HCI Innovations and Experiments (Examples)– MS Hololens, – Magic Leap, – jibo, – sixthsense, – Oculus VR, – NVIDIA AR/VR, PX, – Intel Real Sense, – Google Project Tango, – Interactive Film, – Gear VR

Sam Siewert 23

Summary – Minute PaperModels for How Humans Perceive, Think and ActModels for Computing (State Machines, PLs, Computability)Models for Interaction (CLI, WIMP, PL, NLP, 3D, Neural)

Goals for HCI and Thoughts?– To Extend Human Capabilities with Computers [Efficiently]– To Emulate How Humans Perceive and Think [for Interaction]– To Create an Artificial Intelligence [to Share Cognitive Load]– Is Human Helping Computer with Task Completion or Computer

Helping Human?– Is HCI Inherently Linked to AI and Cognitive Psychology?

Sam Siewert 24

AI, HCI, Cognition Guru Opinions– Numerous Industry Gurus have concern about “Strong AI”

intelligence in computing > human for generalized reasoning (compared to domain specific computing - e.g. numerical methods)AI without HCI could be dangerous (e.g. M2M AI)Singularity [General Intelligence of computers will match and then exceed human] - Nick Bostrom (Oxford FHI), Vernor Vinge, Ray Kurzweil, John Von NeumannElon Musk (OpenAI, Neuralink), Bill Gates (Concerns, but Optimistic), Bill Joy (Sun founder - “Why Future Does not Need Us”)

– Alternative view from Gurus is that Strong AI is a goodwill vastly improve human quality of life and trustworthy M2M and HCI AI will benefit humans overallRay Kurzweil, Neo-Luddites and Age of Spiritual MachinesGoogle - AI, benefits to everyone, Apple Machine Learning

– View that Strong AI is not Possible and therefore of no concernJohn Searle, Colin McGinn, Roger Penrose [Emperor’s New Mind]Computers will never have general intelligence and simply are a prosthetic or instrumentation like a microscope, telescope or any other machine

Sam Siewert 25

Student Responses - HCI-AI No LinkComputers are for “task automation to complete tasks more efficiently than humans” … “the Computer” … “is helping the human” and the human is “directing or commanding” the computer. No link between HCI and AI and Cognition - can be machine to machine, “Although it is not often that AI’s are designed without some inherently linked HCI”.

“relationship between human and machine is symbiotic” … and computers “perform specific tasks assigned” … where “humans are needed to develop” … so “HCI is not linked to AI” … and “few HCIs” … “have capability of learning”

“AI doesn’t necessarily emulate how humans perceive and think” … “AI can be programmed” and … “will have a predictable ‘thought’ process” … whereas humans are “unpredictable”

Sam Siewert 26

Student Responses - HCI-AI LinkedComputers are “built to help the human with task completion” and requires …”efficient and intuitive HCI” with “many methods based in AI and cognitive psychology”

“using AI to share Cognitive Loads” … “good in some situations” … but, can lack “empathy”, e.g. “replacing airline pilots with AI” … “removes the interactions between the passengers and pilots” … so replacing humans is “a balance of competency and emotion”

AI and Cognitive research helps us know “how humans are wired” … “it’s both as to who is helping who” … computers complete “tasks faster than a human ever could”

“Share Cognitive Load” … “definitely a goal of HCI” … “humans are helping the computers so that they can get to a point where they can help humans”

Sam Siewert 27

New Interaction ConceptsAll Interesting, but Need Good Evaluation– Entertainment, Work, Safety, Automation, Other– Computers to Assist Humans and Improve Experience

Ted Talk on Minority Report Style UIPranav Mistry Ted Talk on InteractionMIT Media Lab – “reinvent—how humans experience, and can be aided by, technology.” - HistoryMIT Media Lab Founder Ted Talk

Next, Week 12 … Augmented RealityWeek 13 … Advanced Avionics and UAV/UASWeek 14 – Quiz, Review for Exam #2, A#6 HelpWeek 15 – Exam #2

Sam Siewert 28

SummaryModels for How Humans Perceive, Think and Act

Models for Computing (State Machines, PLs, Computability)

Models for Interaction (CLI, WIMP, PL, NLP, 3D, Neural)

Goals for HCI and Thoughts?– To Extend Human Capabilities with Computers [Efficiently]– To Emulate How Humans Perceive and Think [for Interaction]– To Create an Artificial Intelligence [to Share Cognitive Load]– Is Human Helping Computer with Task Completion or Computer

Helping Human?– Is HCI Inherently Linked to AI and Cognitive Psychology?

Sam Siewert 29