2020 - INFOMMMI - 1 Intro - INFOMM… · Haptics: Enhanced Tactile Feedback Using a Mobile and a...

Multimodal Interaction, Feb 5, 2020

IntroductionWolfgang Hürst

Contact: [email protected], http://people.cs.uu.nl/huerst 1

Contact information

Lectures 1-4, guest lecture & examPeter Werkhoven, UU & [email protected]

Lectures 5-7, exam & projectWolfgang Hürst, UU, Multimedia [email protected], BBG 480(contact me also for administrational questions)

When you email us, start subject with [INFOMMMI]

2/5/20 W. Hürst, http://people.cs.uu.nl/huerst 2

http://wxs.nl

http://uu.nl

IMPORTANT!

Make sure to regularly check your student email, since some important information, last minute

changes, etc. might be sent by email.

It is also a good idea to keep an eye on the “news” page of the course’s website. And refer to the

schedule the for course dates.


Multimodal interaction

Course IntroductionWhat is it all about?


How are we interacting with computers?


KEYBOARD, MOUSE

GRAPHICAL USER INTERFACES

(MULTI) TOUCH

SPEECH

GESTURES & MOTION

DIFFERENT DISPLAYS

Computer usage – from data to experience


1D 3D

Punch cards & cables, …

data information & communication

experience

Touch (& mouse) gestures, …

CLI (Command Line Interface)

Keyboard & mouse

Trackpads, touchscreens, …

Body motions / gestures, tangible UIs, …

Speech, …

Mainframes & workstations PCs & internet Mobile devices,

HMDs, projective displays, …

GUI (Graphical User Interface)

From to to

2D


This requires• Looking into different technologies• Looking into human aspects

(e.g. perception of visual, tactile, and acoustic feedback)

What modalities make sense?And when? And why?

Goal of this course: Learn about different modalities for interaction and when and how to use them.

Overview(lectures)

Human perception• Hearing• Vision• Touch• Multi-sensory• Synaesthesia

Virtual reality• Applications• Technology• Interaction• Navigation• Mobile VR

Augmented reality• Technology• Human factors• Interaction

Lectures are complemented with papers for mandatory reading.Make sure to read them in time, so you have enough time for the project.

Example for human perception …

2/5/20 9W. Hürst, http://people.cs.uu.nl/huerst

Source: National Geographic – Brain Gameshttp://natgeotv.com/nl/brain-games/videos/rubberen-hand

http://natgeotv.com/nl/brain-games/videos/rubberen-hand

Integration of visual and tactile stimuli in a stereoscopic virtual environment

Goal: investigate perception of incongruencies in location between tactile and visual stimuli

Lecture Project Recommended actions

Week 1 IntroductionLecture 1 (vision)

Papers 1, 5

Week 2 Lecture 2 (sound, …)Lecture 3 (VR) Deadline for forming groups

Papers 2, 3, 4& forming project groups

Week 3Lecture 4 (other) Deadline pitch slides: Feb 21, 23:59

Papers 6, 7, 8 & work on project ideas, pitches, plan

Week 4 Project pitchesGuest lecture Deadline project plan: Feb 28, 23:59

Pitch ideas & make project plan

Week 5Lecture 5 (AR) Deadline presentation slides: Mar 6, 23:59

AR papers & setup and implementation

Week 6 Present. (interm.)Lecture 6 (AR) Deadline initial report: Mar 13, 23:59


Week 7Lecture 7 (AR)

Sticky “testing day”: Mar 19Deadline peer review: Mar 20, 23:59

AR papers & experiments

Week 8Deadline project report: Mar 27, 23:59

Analysis & report / paper writing

Week 9 Presentation (final) & group meetings

Deadline presentation slides: Mar 31, 23:59

Examweek

Exam

GradingGRADING PASSING

EXAM (lectures 1-4, guest, lectures 5-8)

Exam grade counts 50% for final grade

Min. 5.0 (before rounding)

PROJECT Project grade counts 50% for final grade

Min. 5.0(before rounding)Min. 5.5 (before rounding)


Participation in all grading-related events (exam, project presentations & meetings) is mandatory.

• Participation in all events (exams, project presentations, meetings)

• Final grade at least 4.0 (after rounding)

Retake: requirements for qualification

PassingNote: Attendance of the lectures is mandatory, too!

However, we are not actively checking it (but expect exam questions that are not covered by the slides and literature).

Attendance in the meetings and presentations related to the project is mandatory and will be checked.


Exceptions may be made if you have a time conflict (but contact me before the event).

• If you failed because your exam average or your overall grade was too low, you can do the retake exam


• If you failed because your grade in the project was too low, you can get a new assignment

Keep in mind that the retake exam is after the 4th quarter (consider this when planning your summer vacation)

Retake – Requirements for passing

• Your retake grade (exam and/or project) will replace the corresponding grade (exam and/or project).

• You pass this course if your new grade fulfills the original requirements


Multimodal interaction

Project IntroductionGeneral remarks

Why a project?

• Hands-on experience is important (esp. when dealing with interaction)

• Many of the learning goals of this projectare motivated by lacks of skills that I identified in small and thesis projects

• Mostly self-dependent work, esp. with respect to• Coming up with research ideas & solutions• Specifying your research goals & questions• Time planning & scheduling


Why a group project?

• Notice the difference between “working self-dependent” and “working alone”

• Lack of self-dependence is also about bringing in own ideas, discuss and defend them, dealing with rather vague specifications

• Important skills for teamwork, which will likely be part of your job (and your thesis project, which usually contributes to a bigger research effort)


How to build the groups?

• Size: 4-5 members should be ideal (but 6 may be okay, too, after discussion)

• You are free to chose your teammates (assignment by me only if necessary)

• Keep in mind your skills and experience when building your teams and specifying your goals (e.g., time is too short to learn a new programming language from scratch)


Use the spreadsheet at http://tinyurl.com/infommmi2020to sign up for a group and find team members.

It is your responsibility to sign up and find a group.

http://tinyurl.com/infommmi2020

Project

Topic & taskWhat you can/have to do



In the lectures, you will learn a lot about different modalities and how we perceive them.

And how we can use technology to create multimodal interaction experiences or how multimodality affects interaction.

Based on this, you should come up with your own idea of a small multimodal interaction research project.


In general, your project will have three phases / parts:1. Setup / implementation2. Experiment3. Analysis

Notice that actual development is preferable but not essential.

Topic & tasks for project

Restrictions:• You are not allowed to hit people with a hammer• No humans or animals should be harmed• No laws should be broken

Any other multimodal experiment should be fine J

Examples for multimodal / multisensory experiments


Localization (small project)

Tactile type (MSc thesis)


Parinya Punpongsanon, Daisuke Iwai, Kosuke Sato. SoftAR: Visually Manipulating Haptic Softness Perception in Spatial Augmented Reality. IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE International Symposium on Mixed and Augmented Reality 2015), Vol. 21, No. 11, PP. 1279-1288.

Examples for multimodal/multisensory experiments


Sungjae Hwang, John Song, and Junghyeon Gim. 2015. Harmonious Haptics: Enhanced Tactile Feedback Using a Mobile and a Wearable Device. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15).




If you have a smartphone & smartwatch (with vibration motors), you could do a similar project

Interesting questions to study:• Different vibration patterns (see paper)• Evaluate the influence of location

(wrist vs. tablet vs. both)

If you use Google Cardboard (and a 2nd

smartphone), you could even do this in VR

You could also use a smartphone for handheld AR (and test, e.g., different feedback such as visuals vs. sound vs. tactile)



Other simple setups include• Visuals & tactile

(e.g., can you influence tactile perception with visuals?)

• Visuals & sound (also on PCs)

Note:• You don’t have to use mobiles,

but can use any tools you like.• You don’t have to do VR/AR,

but anything involving multimodality is ok.• You don’t have to use multiple modalities,

but a test with one modality is also fine, if it fits to the course’s topic.

The major criteria is that you demonstrate that you learned something (useful) from the course.


In previous years, one group investigated the impact of airflow on a roller coaster VR experience.

Notice that this is an example for an experiment that did not require any actual development.


One group tested the influence of audio (e.g., scary vs. happy background music) in a simple VR environment (floating cubes).

Notice that this example required some rather simple development (i.e., a well-suited topic for beginners)


One team used vibration motors hooked up to an Arduino to study frictional effects in VR and their impact on presence


And one group put various sensors (distance measures, tactile and auditory feedback) into a hat to provide navigation aid for visually impaired.

Evaluation in a setup maze environment with blindfolded participants and combinations of none, auditory, and tactile feedback.

Usage of tools (hardware & software)

You are welcome to use any hardware you have available (but make sure to guarantee availability during the whole project)

You are welcome to use existing software (but make sure to specify it in your project report)

We might be able to provide you with: • HTC Vive VR HMDs (max. 3 are available)• Two old Oculus Rift VR HMDs (1 DK1 and 1 DK2)• Meta 1 and Meta 2 AR HMDs

Note that you might need to share them with some MSc students and there’s a lack of computers and rooms (sorry for that).

You are encouraged to take advantage of this, but not required (e.g., non-VR projects are fine, too).

We will discuss hardware assignments and availability after your pitches.




Papers 1, 5


Papers 2, 3, 4 &forming project groups

Week 3Lecture 4 (other) Deadline slides: Feb 21, 23:59

Papers 6, 7, 8 &work on proj. ideas, pitches, plan

Week 4 Project pitchesGuest lecture Deadline proj. plan: Feb 28, 23:59


Week 5Lecture 5 (AR) Deadline slides: Mar 6, 23:59


Week 6 Present. (interm.)Lecture 6 (AR) Deadline initial report: Mar 13,

23:59



Sticky “testing day”: Mar 19Deadline review: Mar 20, 23:59

AR papers &experiments

Week 8Deadline report: Mar 27, 23:59



Deadline slides: Mar 31, 23:59

Examweek

Exam

Complaint in a previous year:


“This is way too little time for a project like this!”




• The course is 7.5 ECTS, which corresponds to 210 hours.

• The project grade counts 50%, so 105 hours.

• Multiply this by number of team members(and subtract some hours for communication overhead).

With good planning, this should be enough to do something great!







• You decide what you do.• It is part of the learning goals to:

– Find an interesting & relevant topic– Find a feasible topic!– Make a good time planning!– And stick to it!

Good planning & topic selection

• It is okay when your study has limitations …… if you can explain them and are aware of the consequences.

• It is okay to (partly) fail because you aimed too high …… if you can explain why and still achieved a learning goal

(and your original goal was realistic at the time)

• Important issues are:– That you demonstrated that you learned something (useful)

from the course– That your plans and study make sense as a whole (even if they

have limitations)

Note: you are allowed to take risks, be creative, go crazy, surprise me with cool ideas!


You can approach this in two ways:• Study a cognitive issue

(but don’t forget to address the practical impact)• Look at an applied/computer engineering problem

(but make sure to address a scientific problem)


But how to get from an ideato a research questionto scientific results?

Project

How to come up with a good research topic?Some general tips & comments


The development of a research idea

Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke Johnson, Larry Christensen, Fig. 3.1, page 64

Research problemThe issue or problem within a broad topic area

Research purposeA statement of the intent or objective of the study

HypothesisA prediction or best guess of the relation that exists among the variables

being investigated

Research questionIn quantitative research it is an interrogative sentence that asks a question

about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue,

or phenomenon to be explored.

Research topicThe broad subject matter area to be investigated

For self study


Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke Johnson, Larry Christensen, Fig. 3.1, page 64

Research problemThe issue or problem within a broad topic area

Research purposeA statement of the intent or objective of the study

HypothesisA prediction or best guess of the relation that exists among the variables

being investigated

Research questionIn quantitative research it is an interrogative sentence that asks a question

about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue,

or phenomenon to be explored.

Research topicThe broad subject matter area to be investigated

Notice that this is from “Educational research” (but it applies to all kinds of empirical and quantitative research)


Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke

Johnson, Larry Christensen, Fig. 3.1, page 64

Research problem

Research purpose

Hypothesis

Research question

Research topic Notice the differences:• From the general context / bigger aim• To a concrete research (sub) problem• To a very concrete research questions

that can be answered with scientific methods

Example:• MSc thesis

Nina Rosa (published at ICMI’15)




Research problem

Research purpose

Hypothesis

Research question

Research topic Research topic:The broad subject matter area to be investigated

Example:We know from literature that incongruent stimuli can change perception. Can we take advantage of such effects to create richer VR experiences by providing incongruent tactile and visual stimuli?




Research problem

Research purpose

Hypothesis

Research question

Research topic Research problemThe issue or problem within a broad topic area

Example:How does perception of tactile feedback change when incongruent visual stimuli are presented simultaneously?In particular, can we use certain visual signals to create a certain tactile perception?




Research problem

Research purpose

Hypothesis

Research question

Research topicResearch purposeA statement of the intent or objective of the study

Example:Investigate if visual stimuli that represent certain characteristics or experiences can be used to modify our perception of certain tactile stimuli accordingly.




Research problem

Research purpose

Hypothesis

Research question

Research topic Research questionIn quantitative research it is an interrogative sentence that asks a question about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue, or phenomenon to be explored.

Example:Is it possible to create the illusion of experiencingdifferent intensities of a certain property (weight or temperature) using a rather simple and unrelated type of touch (vibrations) together with compelling, type-related visuals (speed/size and color, respectively)?




Research problem

Research purpose

Hypothesis

Research question

Research topic Terminology:• Independent variables = the input or

cause (i.e., what we control)• Dependent variables = the output or

outcome whose variation is being studied (i.e., what we measure)

The purpose of an experiment is generally to verify how the latter depend on the former.Ideally, they are reflected in your research question.

Example:• Independent variables:

• Color (red, gray, blue)• Falling speed and size

• Dependent variable:• Experienced intensity of properties




Research problem

Research purpose

Hypothesis

Research question

Research topicHypothesisA prediction or best guess of the relation that exists among the variables being investigated

Example:• We expect that visuals commonly

associated with weight (speed/size) lead to an increased perception of tactile intensity

• We expect that visuals commonly associated with temperature (color) lead to an increased perception of tactile intensity

But what if we can’t make such an educated guess?




Research problem

Research purpose

Hypothesis

Research question

Research topicHypothesisA prediction or best guess of the relation that exists among the variables being investigated

Terminology• Null-hypothesis: assumes no

relationship or significant difference between groups on one variable

• Directional hypothesis: predicts an outcome in one direction based on prior literature or other reasoning

• Nondirectional hypothesis: similar to directional hypothesis, but the exact form of difference (e.g., higher, lower, more, less) is not specified




Research problem

Research purpose

Hypothesis

Research question

Research topic

Be careful! There is no general agreement on the terms on the left!

People use the same term in different ways.

But the basic procedure (from “high level/big picture” to “concretely verifiable statement”) is generally the same.




Research problem

Research purpose

Hypothesis

Research question

Research topic Global aim / general goal

Related research problem(s) that need to be solved to achieve this

(big) goal

Related research questions that need to be answered to solve these

problems

Concrete objectives that need to be addressed to answer them

This is the terminology that I normally use.


Global aim / general goal

Related research problem(s) that need to be solved to achieve this

(big) goal

Related research questions that need to be answered to solve these

problems

Concrete objectives that need to be addressed to answer them

This is the terminology that I normally use.

And don’t forget that the sub-problem

that you address must be relevant!

For your projectWhat (general) problem are you approaching?What concrete sub-problem will you address?How are you going to verify your ideas/solutions?

Three slides (strict; plus one intro & one closing).

1. Problem (general goal and concrete sub problem)

2. Solution (your idea to address the sub problem and/or the research question you want to answer, related hypothesis, and objectives)

3. Methodology (your planned approach to verify your results or proof that your solution works)

Time limit is 5 minutes (& will be strictly enforced).

Recommendation for brainstorming: Try answering these questions

Project pitch (Feb 24):


Project pitches1. Problem2. Solution3. Methodology


Note: your problem and solution specification depend on your goal.

Most of you will be doing some sort of an empirical evaluation.

It is okay though to address a more engineering problem or an HCI-related issue.

While they use different approaches, they usually follow the same ”top-down” structure illustrated before.

And they also rely on objective evaluations (which generally means your problem solution should be measurable and results should be repeatable).

For your projectWhat (general) problem are you approaching?What concrete sub-problem will you address?How are you going to verify your ideas/solutions?

Three slides (strict; plus one intro & one closing).

1. Problem (general goal and concrete sub problem)

2. Solution (your idea to address the sub problem and/or the research question you want to answer, related hypothesis, and objectives)

3. Methodology (your planned approach to verify your results or proof that your solution works)

Time limit is 5 minutes (& will be strictly enforced).

Recommendation for brainstorming: Try answering these questions

See also info on the website!

Project pitch (Feb 24):



“Soft” goals (for project):• Self-dependent work, e.g.

– Coming up with own ideas– Being able to specify them into research questions– Making realistic time planning and scheduling

• Teamwork

Learning goals / objectives of the course (project):

Projects: after successful completion of this part of the course ...• Students can apply basic techniques of multisensory interfaces to

create a multimodal interaction, e.g., for control of a game (demonstrated by the related project work).

• Students know the advantages and disadvantages of using different media types and modalities depending on different contexts, conditions, and applications (demonstrated by the related project work).

• Ideally/normally, you will get a team grade– Exceptions apply for particularly motivated or lazy students

– Keep a logbook tracking your hours and activities!– You only have to report your total hours and major

contributions/activities in the end (see report template)– Logbook is needed if above exceptions need to be applied

– Note: You won’t get punished for other people’s failures(but you might get punished for badly dealing with them)

– Better grades than the group grade are possible (e.g., for cum laude students), but YOU must convince me why you deserve it

You find more info on the grading in the document about the project report (see website). Read it now!


Some comments on grading



Papers 1, 5


Papers 2, 3, 4 &forming project groups

Week 3Lecture 4 (other) Deadline slides: Feb 21, 23:59

Papers 6, 7, 8 &work on proj. ideas, pitches, plan

Week 4 Project pitchesGuest lecture Deadline proj. plan: Feb 28, 23:59


Week 5Lecture 5 (AR) Deadline slides: Mar 6, 23:59


Week 6 Present. (interm.)Lecture 6 (AR) Deadline initial report: Mar 13,

23:59



Sticky “testing day”: Mar 19Deadline review: Mar 20, 23:59

AR papers &experiments

Week 8Deadline report: Mar 27, 23:59



Deadline slides: Mar 31, 23:59

Examweek

Exam

1. Read info on the website (all documents; the ones under “HELPFUL LITERATURE” are optional, but might come in handy)

2. Form groups (via spreadsheets); also fill out all the fields below (for scheduling)

3. Brainstorm ideas while following the lectures & reading papers4. Start preparing project pitches (for Feb 24)

Note that they don’t have to be perfect and can contain open questions to ask the audience!

5. Start working on project plans (due Feb 28)You find a template and further info on the website.

And most importantly: have fun and enjoy the course!


Summary of next steps

If you have questions, email me ([email protected]).

Date post:	20-Aug-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times

2020 - INFOMMMI - 1 Intro - INFOMM… · Haptics: Enhanced Tactile Feedback Using a Mobile and a...

Documents