Computer-based Experiments:Computer-based Experiments:ObstaclesObstacles

Stephanie BryantUniversity of South Florida

Note: See Bryant, Hunton and Stone, BRIA 2004 for complete references

Obstacles—OverviewObstacles—Overview

Technology Skill NeededTechnology Skill Needed Threats to Internal ValidityThreats to Internal Validity Getting ParticipantsGetting Participants

Obstacles (Con’d)Obstacles (Con’d)

Technology Skills NeededTechnology Skills Needed ““Proficiency” in software or programmingProficiency” in software or programming

Develop Using a Scripting Language or Applications Develop Using a Scripting Language or Applications SoftwareSoftware

Applications Software for Web ExperimentsApplications Software for Web ExperimentsExample software packages: RAOSoft, Inquisite, PsychExpsExample software packages: RAOSoft, Inquisite, PsychExps

More expensive software, cheaper development & More expensive software, cheaper development & maintenance costs? Easier to use, Features = those built maintenance costs? Easier to use, Features = those built into the softwareinto the software

Scripting languages: Scripting languages:

Examples: Cold fusion, PHP, JSP (java server pages), Examples: Cold fusion, PHP, JSP (java server pages), CGI (common gateway interface)CGI (common gateway interface)

Software is cheap or free, higher development & Software is cheap or free, higher development & maintenance costs?, difficult for non-programmers, More maintenance costs?, difficult for non-programmers, More features, more customizablefeatures, more customizable

Combine Scripting Languages & applications softwareCombine Scripting Languages & applications software

Tools for an Computer-based Tools for an Computer-based ExperimentsExperiments

Applications Software: Raosoft Applications Software: Raosoft Products (Products (Ezsurvey, Survey win,

Interform) Difficulty index (1 = hard,10 = easy): 8Difficulty index (1 = hard,10 = easy): 8Do not provide all the functionalitiesDo not provide all the functionalities

No randomization, response dependent No randomization, response dependent questions (I.e., only straight surveys) questions (I.e., only straight surveys) Limited formatting capabilitiesLimited formatting capabilities

Expensive – no educational prices ($1,500 - Expensive – no educational prices ($1,500 - $10,000)$10,000)SurveyMonkey.com - $19.95/monthSurveyMonkey.com - $19.95/month

SurveyMonkey.comSurveyMonkey.com

Applications Software: InquisiteApplications Software: Inquisite

Difficulty index (1 = hard,10 = easy):8Difficulty index (1 = hard,10 = easy):8 Expensive ($10,000) Supports most of Expensive ($10,000) Supports most of

functionalitiesfunctionalitiesTo support all desired functionalities requires To support all desired functionalities requires

Software Development Kit (SDK) for complex Software Development Kit (SDK) for complex applications ($2,000 but may be available soon applications ($2,000 but may be available soon for free)for free)

Applications Software: PsychExpsApplications Software: PsychExps

PsychExperiments Web site created and PsychExperiments Web site created and maintained by the Univ. of Mississippi maintained by the Univ. of Mississippi Psychology professor Ken McGraw.Psychology professor Ken McGraw.

““Collaboratory”Collaboratory” http://psychexps.olemiss.edu/http://psychexps.olemiss.edu/ Free!Free! Requires that user download & install applications Requires that user download & install applications

software software Many existing scripts (e.g., randomization)Many existing scripts (e.g., randomization)

Psychexps Home PagePsychexps Home Page

Psychexps (Con’d)Psychexps (Con’d)

Current Experiments on PsychexpsCurrent Experiments on Psychexps

Obstacles (Con’d)Obstacles (Con’d)

Big learning curves involvedBig learning curves involved On-campus support sometimes availableOn-campus support sometimes available Can hire programmers/graduate students to Can hire programmers/graduate students to

help with programminghelp with programming

Obstacles (Con’d)Obstacles (Con’d)

Internal Validity Considerations: Internal Validity Considerations: Statistical Conclusion ValidityStatistical Conclusion ValidityInternal ValidityInternal ValidityConstruct ValidityConstruct ValidityExternal ValidityExternal Validity

Statistical Conclusion ValidityStatistical Conclusion Validity(The extent to which two variables can be said to co-vary)(The extent to which two variables can be said to co-vary)

Increased sample size and statistical power Increased sample size and statistical power (e.g., Ayers, (e.g., Ayers, Cloyd et al.)Cloyd et al.) Web to recruit participants!Web to recruit participants!

Decreased or eliminated data entry errors Decreased or eliminated data entry errors Capture data directly into databaseCapture data directly into database

Increased variability in experimental settings Increased variability in experimental settings Difficult to control in Web experimentsDifficult to control in Web experiments People complete experiments in their own (“natural”) settings with People complete experiments in their own (“natural”) settings with

various types of computer configurations (browsers, hardware)various types of computer configurations (browsers, hardware) McGraw et al (2000) note that WE noise is compensated for by McGraw et al (2000) note that WE noise is compensated for by

large sample sizeslarge sample sizes System Downtime System Downtime Software Coding Errors Software Coding Errors (e.g., Barrick, 01, Hodge 01) (e.g., Barrick, 01, Hodge 01)

Internal ValidityInternal Validity(Correlation or Causation?)(Correlation or Causation?)

Decreased potential diffusion of treatment Decreased potential diffusion of treatment Unlikely that participants will learn information intended for

one treatment group and not another. Increased participant drop-out rates across Increased participant drop-out rates across

treatments treatments A higher drop-out rate among Web vs. laboratory experiments

could create a participant self-selection effect that makes causal inferences problematic.

Mitigate by placing requests for personal information and Mitigate by placing requests for personal information and monetary rewards at the beginning of the experiment (Frick et al monetary rewards at the beginning of the experiment (Frick et al 1999) and McGraw et al. (2000).1999) and McGraw et al. (2000).

Completion rate approached 86% when some type of monetary Completion rate approached 86% when some type of monetary reward was offered (Musch and Reips 2000)reward was offered (Musch and Reips 2000)

Internal ValidityInternal Validity(Correlation or Causation?)(Correlation or Causation?)

Controlling “cheatingControlling “cheating” Multiple submissions by a single participantIdentification by email address, logon ID, password, or

IP address Randomization (A control) Randomization (A control)

Computer scripts available for randomly assigning Computer scripts available for randomly assigning participants to conditionsparticipants to conditions

Complete scripts published in Baron and Siepmann Complete scripts published in Baron and Siepmann (2000 247) and Birnbaum (2001, 210-212)(2000 247) and Birnbaum (2001, 210-212)

Construct ValidityConstruct Validity(Generalizability from observations(Generalizability from observations

to higher-order constructsto higher-order constructs))

Decreased demand effects & other Decreased demand effects & other experimenter influences experimenter influences Rosenthal (66 & 76), Pany (87)Rosenthal (66 & 76), Pany (87)

Decreased participant evaluation Decreased participant evaluation apprehension apprehension Rosenberg (69)Rosenberg (69)““Naturalism” of setting decreases? Naturalism” of setting decreases?

Getting ParticipantsGetting ParticipantsWeb-based ExperimentsWeb-based Experiments

Explosion of WWW Use Explosion of WWW Use

172 million 172 million computers linked to computers linked to WWWWWW

90% of CPAs conduct 90% of CPAs conduct internet researchinternet research

60% of US population 60% of US population has WWW accesshas WWW access

Getting ParticipantsGetting Participants

Internet Participant SolicitationInternet Participant Solicitation

• Benefits

─ Large sample sizes (power) possibleLarge sample sizes (power) possible

─ Availability of diverse, world-wide Availability of diverse, world-wide populationspopulations

─ Interactive, multi-participant responsesInteractive, multi-participant responses

─ Real-time randomization of question orderReal-time randomization of question order

─ Response dependent questionsResponse dependent questions (branch and bound)(branch and bound)

─ AuthenticationAuthentication and authorization and authorization

─ MultimediaMultimedia (e.g., graphics, sound)(e.g., graphics, sound)

─ On-screen clockOn-screen clock

Getting ParticipantsGetting Participants

Web-basedWeb-basedPost notices in places where your target Post notices in places where your target

population might be likely to visitpopulation might be likely to visitAccess ListServsAccess ListServs

PC-basedPC-basedStudent involvement requirement??Student involvement requirement??USF ProcessUSF Process

USF ProcessUSF Process

Mandatory participation in one experiment Mandatory participation in one experiment per semesterper semester

Experimentrix site used to manageExperimentrix site used to manage https://experimetrix2.com/soa/https://experimetrix2.com/soa/

https://Experimetrix.com/soahttps://Experimetrix.com/soa

Experimetrix SignupExperimetrix Signup

Experimenter ReportExperimenter Report

A Final Caveat: What Can Go A Final Caveat: What Can Go Wrong…WillWrong…Will

Cynical, but realisticCynical, but realistic Plan carefullyPlan carefully Develop contingency plansDevelop contingency plans Consider cost-benefitConsider cost-benefit Greatest potential for BAR Web Greatest potential for BAR Web

experiments is as yet unrealizedexperiments is as yet unrealized Biggest hurdle is required knowledge, but Biggest hurdle is required knowledge, but

this can be overcomethis can be overcome