Sophistication in Research in Marketing

Donald R. LehmannLeigh McAlisterRichard Staelin

“Ideas that Matter” Pre-Conference Summer AMA 2010

Review process• Execution quality• Idea quality

Submitted papers

Faculty

Published papers

Promotion &Tenure review

Human capitalKnowledge from other fields

PhD trainingHiring

The Research SystemAnd Feedback Loops

Solid Lines = Direct EffectsDash Lines = Feedback Effects

Review process• Execution quality• Idea quality

Submitted papers

Faculty

Published papers

Promotion &Tenure review

Human capitalKnowledge from other fields

PhD trainingHiring

Lynch

Old Days: More AspirationFor Conceptual Papers

Today: Don’t Even ConsiderIt Before Tenure

Review process• Execution quality• Idea quality

Submitted papers

Faculty

Published papers

Promotion &Tenure review

Human capitalKnowledge from other fields

PhD trainingHiring

Lynch

Need more papersfor promotion

Count instead ofread

Review process• Execution quality• Idea quality

Submitted papers

Faculty

Published papers

Promotion &Tenure review

Human capitalKnowledge from other fields

PhD trainingHiring

Lynch

Process impacts PhD Training

Nature of papersPhD students write

Review process• Execution quality• Idea quality

Submitted papers

Faculty

Published papers

Promotion &Tenure review

Human capitalKnowledge from other fields

PhD trainingHiring

Lynch

Journals publishingmore papers More authors More papers/author

Review process• Execution quality• Idea quality

Submitted papers

Faculty

Published papers

Promotion &Tenure review

Human capitalKnowledge from other fields

PhD trainingHiring

LMS

Drawing scholars fromother fields New methods New theories Less interest in marketing

Review process• Execution quality• Idea quality

Submitted papers

Faculty

Published papers

Promotion &Tenure review

Human capitalKnowledge from other fields

PhD trainingHiring

LMS

Outside scholars drivetechnical sophisticationarms race

Not needed/helpfulunderstandingmarketing

Review process• Execution quality• Idea quality

Submitted papers

Faculty

Published papers

Promotion &Tenure review

Human capitalKnowledge from other fields

PhD trainingHiring

LMS

Accelerates Elison-identified: “Executiondrives out Ideaquality”

Yadav (2010) shift fromconceptual articles

Reibstein/Day/Wind (2009)shift away from relevance

Survey of Published Authors

• JM, JMR, JCR, MktSci; Past 5 years• 510 sent out• 56% response rate– No follow up notification

• Target Journal– 20% JM– 40% JCR– 20% JMR– 20% Marketing Science

What makes a good review?

Useful Reviews• Enhance managerial

relevance• Provide theoretical

justification• Rule out alternative

explanations

• ≈ Idea Quality

Not Useful Reviews• Use a different model• Use a more complex

method• Alter the thrust of the paper

• ≈ Execution Quality

Asked to increase managerial relevance….

• 45% said it resulted in new insight

• 41% said it improved the focus of the paper

Asked to use more complex method….

• 80% said it had no impact on paper’s conclusions

• 38% said it made the paper harder to read

• People said they got this request 50% of the time

Old Fart-itis? No.

Responses didn’t differ by cohort, exceptIn the field 5 years or less

Less frequent requests for more complex modelFewer conflicting requests from reviewers

In the field…

5 years or less 22%

6-10 years 23%

11-20 years 26%

More than 20 years 24%

What to do?

• Editors– Tell reviewers to rebalance method and idea– Back up policy with editorial decisions

• Reviewers– Interesting– Not wrong

What to do?

• More complex method is appropriate if:– Assumptions are met– Method is focus of research– Method changes the result– Benefit > extra communication cost– Method is described simply