Data Collection

In the Beginning

Define Constructs (what is the domain?)Therapy effectivenessIntegrity tests

Research QuestionWhat’s the average? Zero? Impact of management (e.g., Brown*)May or may not be summary of a literature

Theoretical Justification of Moderators

Pick ONE study type (e.g., experiment, correlational study) or pick all and analyze separately.

*Brown, S. (1981). Validity generalization and situational moderation in the life insurance industry. Journal of Applied Psychology, 66, 664-670.

Sources of Data (ES) 1

Electronic DatabasesPsych info, APA onlineSSCI, Web of Science, PubMedGoogle scholar*; What others?

May hire a reference librarian

Conference Proceedings & Abstracts

DissertationsLocalDistant

Government Docs & Agencies

*So far as I know, you cannot use Boolean AND or NOT in Google Scholar, so you get too many returns.

Sources of Data (2)

Hand searches of the literatureRefs in booksRefs in articlesFlip thru journals

People in the area (write, email, drink)Use your contacts (faculty, friends, etc.)Butter up

Review Articles (used to be more)

Unpublished manuscripts (prepubs, too)

Eligibility (1)Country

Language

Measure (specific)

NEO vs Big Five

Beck vs CESD

Measure (general)

Reliability

N of items

Participants (e.g., female cancer survivors; bankers; no students; employed 20 + hr/ week, etc.)

Publications (what type)

Published vs. not

The criteria are your choice. But you may have to defend that choice. Do you have good reasons?

Eligibility (2)Time frame

Gains over time

Time after treatment (e.g., smoking)

Required information

Effect Size

Moderator

Other

You can contact authors for missing info

Study Design

Control group

Repeated Measures

Random Assignment

Specific controls (e.g., dismantling)

Methodological Quality? Code for features, not overall quality

Database 1

Keep records of the search process and the elimination process to create a flow chart (replication). Example of flowchart soon.

Create a database to hold the effect sizes, data needed to compute them, and potential moderators

Excel is good for a database, but you can use whatever you like

Database 2

Prepare for reliability of coding – copy code book and database

At least 2 coders who will code ALL the variables independently for at least a sample of studies

Create system to track dependencies (effect sizes from the same study, etc.)

Code Author and Date of Publication separately for cumulative analysis by date

Example Flow Chart

Example Search Setup

Note. We sort the documents into bins. Also see the files ‘SearchAurah.xlsx’ and ‘SearchProcessmb.docx’ for examples. The Excel file is for the flow chart. It has no space for coding ES.