HUMAN-COMPUTER INTERACTIONMETHODOLOGY MATTERS
Professor Bilge MutluComputer Sciences, Psychology, & Industrial and Systems Engineering
University of Wisconsin–Madison
CS/Psych-770 Human-Computer Interaction
Doing research is the systematic use of some set of theoretical and empirical tools to try to increase our understanding of
some set of phenomena or events.
BASIC FEATURES
All research brings together:
Content
Ideas
Techniques and procedures
THREE DOMAINS OF RESEARCH
Substantive
Actors and context
Conceptual domain
Behavior or relations
Methodological domain
Modes and techniques
TECHNIQUES
Techniques for measurement
Techniques for manipulation
Techniques for controlling impact
MEASUREMENT
Measuring some feature of a research situation
MANIPULATION
Systematically varying components of a system
Giving instruction
Imposing constraints
Selecting materials
Giving feedback
Using confederates
CONTROL
Controlling the impact of various extraneous features of the situation
Experimental control
Statistical control
Distributing impact (e.g., randomization)
COMPARISON
Independent variables
Dependent variables
The research process, like a three-legged stool, always depends on
materials from all three domains—content, ideas, and techniques.
LIMITATIONS
Methods pose opportunities and limitations;
Might have weaknesses that limit evidence
Can offset weaknesses by using multiple methods
CHOOSING A SETTING
Three key considerations
Generalizability
Precision of measurement
Realism of the situation
Seek to maximize all three
Not attainable but we do our best
RESEARCH STRATEGIES
Field strategies
Experimental strategies
Respondent strategies
Theoretical strategies
oftheothertwo.Forexample:
Thethings
youcandototrytoincreasetheprecision
i:O'"'i"!:l
....
.t;:so 0';'
U1
with
which
youcanmeasure
behavior
andcontrolrelatedvariables
(B)(forexample,
"$:
...~~en
conductingacarefullyconLl'olledlaboratoryexperiment)willintrudeuponthesituation
om>
;::~a
w~"
'.andreduce
its"naturalness"
orrealism
(thatis,reduce
C),andwill
also
reduce
the
IIIIII
~JJ
(:)~
3:rangeofactors
(A)towhom
thefmdings
canbe
generalized.
Conversely,
thethings
you
JJ~(j)
3'CD
:1C'\
01/1
91u
CD~CD
ID"'C
G)
ID::J
o0
candototry
tokeephigh
realism
ofcontext(C)(forexample,conductingafieldstudy
=_.~
0'::JI:
...~!e.III
...C,
IDII)
inanaturalsituation)willreduceboththerangeofpopulationstowhichyourresults
0=~!!!
~~--
.-+::IN
!!!.
Ro~ID
-ID
;J'canbeapplied(A)andtheprecisionoftheinformationyougenerate(B).Asathird
g;;:r
m::J
~~~0
::;--
example,thethingsyoucandototrytoestablish
ahighdegreeofgeneralizabilityover
::I:
ID=
~CD
~-a
actors
(A)(forexample,
conducting
awell-designed
sample
survey)will
reduce
real-
::J.
0iii'
~~1/1
ism(C)by
obtaining
themeasures
outofcontext,
andwill
reduce
precision
(B)both
byhaving
measures
ofonly
alimited
number
ofbehaviors,
andby
failing
tocontroloroth-
erwise
take
into
accountextraneous
factorsthatmay
affect
results.
me..
Youcanappreciate
thisdilemmabetterby
examiningsomeofthemajor
research
-I: Cc,
c'(Q
strategiesusedinthebehavioral
andsocialsciences.Figure2showsasetofeightalter-
'<3
mm
CDnativeresearch
strategies,orsettingsforgatheringresearch
information.
Inthatfigure,
I:D!
a<3
theeightstrategiesareshownaslyinginacirculararrangementinrelationtotwo
CD"'C
'<(jj'
underlyingdimensions:thedegree
towhich
thesetting
usedinthestrategyisuniversal
mr
orabstractvs.particular
orconcrete;andthedegree
towhich
thestrategyinvolvespro-
)(& -go
ceduresthatareobtrusive,vs.proceduresthatareunobtrusive,
with
respecttotheongo-
::J.iiI
ier
inghuman
systems(theactor-behavior-context
units)thataretobetheobjectofstudy.
a-<
ThefOUTstrategies
ontherightsideofthecircleinvolvefairlyconcrete
orparticularis-
ticsettings;thefourontheleftsideusefairlyuniversalorabstractsettings.Theproce-
duresusedinthefourstrategiesinthelowerhalfofthecirclecanbefairlyunobtrusive.
Thefourstrategiesinthetophalfofthecirclenecessarily
useprocedures
thatarefair-
mo
lyobtrusive,
thatis,theydisturb
theongoinghuman
systems(theactor-behavior-con-
~r ~E."'C
textunits)thatarebeingstudied.
!!!.C
ci"CD
Figure2alsoshowswhere,amongthestrategies,eachofthethreedesired
features,
::J'"
orcriteriaisatitsmaximum.CriterionA,generalizabilitywithrespecttothepopula-
m-n
[~tionofActors,ispotentially
maximizedinthesamplesurveyandinformaltheory.
,<c,
Criteriona,precisionwithrespecttomeasurementandcontrolofbehaviors,ispoten-
m"
tiallyatitsmaximum
inthelaboratoryexperimentandinjudgmentstudies.Criterion
)(-.
-g~
C,realism
ofcontext,ispotentiallyatitsmaximum
inthefieldstudy;Thegeometryof
::I.
.3
>figure2-1emphasizes
thedilemmajustdiscussed,
namely:
strategiesthatmaximize
CD ar::r !4.
oneofthesearefarfromthemaximum
pointfortheothertwo.Theverysamechanges
iiI~in researchproceduresthatwouldletyoumovetowardthemaximum
ofanyone
of"
thesecriteria-A,a,orC-atthesametimewouldmoveyouawayfromthemaxi-
...~o
Ii0
c'cmum
pointoftheothertwo.
Itisnotpossible,
inprinciple,
tomaximize
allthree
crite-
0mID
::J_c,
();i1
riasim
ultaneously.Thus,anyone
researchstrategyislimitedinwhatitcanachieve.
iii-::J
;-CD_
(Q-
Research
done
byanysingle
strategy
isflawed,
although
thevarious
strate!";"s
are
iii"
1/1
flawed
indifferent
ways.
QUESTIONS?
COMPARISON TECHNIQUES
Correlation
Systematic covariation in the values of two properties of a system
Linear or nonlinear
Difference
Variance in a variable across different levels of another variable
RANDOMIZATION
Random assignment of cases to conditions
A “true” experiment
Sampling technique plays an important role
VALIDITY
Internal validity
Construct validity
External validity
MEASUREMENTS
Self-reports
Observations
By visible or hidden observers
Archival records
Private or public
Trace records
MANIPULATIONSelection
Varying the population across conditions
Direct intervention
Varying the structure of or processes in a system
Indirect inductions
Evoking varied responses
SUMMARY
Methods dictate the results the researcher will obtain
This is why it is extremely important to report all details of your method
It is impossible to maximize all desirable features of a method
Hence we have “limitations” sections in our papers
You need to interpret your results in the light of other related results
This is why we include relevant background in our papers and interpret our results in the light of the results from this background
QUESTIONS?
THANKS!Professor Bilge Mutlu
Computer Sciences, Psychology, & Industrial and Systems EngineeringUniversity of Wisconsin–Madison
CS/Psych-770 Human-Computer Interaction | Fall 2010