EDITORIAL DECISIONS & DETERMINANTS OF IMPACT

Ayeh Bandeh-AhmadiDepartment of EconomicsUniversity of Maryland &hypochondri.ac

Goals/Outcomes of This Study

• Better design of innovative organizations• Case study of Knowledge Commons use• Insights for designing knowledge commons• Development of broader set of metrics

including text-based• Beyond publications, applications to

– grants, prizes, fellowships, society elections

Background• Journal publications are broadly used for:

– job placement, tenure, grants, prizes

• Landscape is changing with:– advent of electronic publications (eg. PLoS)– financial pressure on publishers– blogs, working papers

• Opportunities to build new institutions and tweak existing ones

More Context“a system in which commercial publishers make profits based on the free

labor of mathematicians and subscription fees from their institutions’ libraries, for a service that has become largely unnecessary”

Signed by 34 mathematicians, February 2012

• Ongoing realities:– Elsevier Boycott– Opportunities for design of Scientific Commons

• Research Questions:– What are incentives for editors, referees, authors, publishers?– What does quality actually mean? Citations?

• Learning experiences with proprietary arrangements• Insights for designing Scientific Commons

Stakeholder Framework• Interactions between stakeholders:

PublishersEditorsRefereesAuthorsLibrariesReaders

Researchers

StudentsProfessorsAcademic OrganizationsResearch Funding AgenciesOthers?

Existing Theories/Related Literature

• Abrevaya & Hamermesh 2009– No evidence of gender bias amongst econ

referees

• Cherkashin et al (Type I/II errors in accept/reject at Journal of International Economics)– Co-editor standards seem to vary significantly– Type I error small (rejected papers less cited)– Type II error large (poorly cited accepted

papers)

Existing Theories/Related Literature

• Ellison (2002)– Theory of editorial incentives & referee

communication explains increasing lags

• Laband (1990)– Survey data characterizes critical referee roles,

editors feeling they must take marginal work

• Labande & Piette (1994)– Peers’ work selected by editors often yield

greater citations

Theory: Factors affecting editorial decisions, referee recommendations,

citations:• Editors select papers to publish based on:

– Fit within journal’s niche– Accuracy– Future Impact– Pressure to fill pages from publishers– Perhaps also personal benefit (citations, etc.)

• What role do referees play?– Providing feedback to improve quality of mediocre

papers, rich information content in referee letters– Evaluating fit, accuracy, future impact, personal benefit

• What drives citations?– Hot topics, famous authors, publication in journals

Data: Editorial Databases

• Manuscript ID• Manuscript Received Date• JEL codes• Country• Co-editor ID• Co-editor Evaluation:

– Summary Reject, No Referee Input

– Summary Reject, Referee Input

– Reject– Withdrawn– Returned for Revision– Conditionally Accepted (Minor

Revisions)

– Accept

• Revision• Manuscript text• Referee review text• Referee ID• Referee Evaluation:

– No Recommendation (i.e. no review submitted)

– Definite Reject– Reject– Weak Revise & Resubmit– Revise & Resubmit– Strong Revise & Resubmit– Accept with Revisions– Accept

Data from Five Journals: 5372 submissions; 2004-2010

Building Meaningful Metrics for Testing Theories

• Textual content of manuscripts– similarity to past accepted articles in journal– Presence of co-editor name in paper references

(first, last versions)– Paper length

• Textual content of referee reviews• Referee scores (past and current)• Editorial scores (past and current)• Citation Commons (IDEAS, Google Scholar)

Textual similarity between submissions

• Edges show relationships between submissions with textual cosine similarity of at least 18%.

• Green nodes are accepted submissions; pink nodes are all other submissions

• Layout using spring-embedded algorithm (Kamada & Kawai, 1988).

Models:

LASSO citation model selection method: Predict citations of journal submissions – based on referee language -- whether they are published or not

Ordered Probit model of referee decision

Ordered Probit model of editorial decision

Referee Language Determinants of Score, Citations

• Little correlation between referee language coefficients on referee score versus eventual citations

0 1000 2000 3000 4000 5000 60000.00

1,000.00

2,000.00

3,000.00

4,000.00

5,000.00

6,000.00

f(x) = 0.00421780677319079 x + 2850.91641816366R² = 1.77898955144862E-05

rank(score_coeffs_pvalue)

rank(c

ite_c

oeff

s_p

valu

e)

0 1000 2000 3000 4000 5000 60000

1000

2000

3000

4000

5000

6000

f(x) = 0.157560363264924 x + 2411.89788350376R² = 0.0248253224057068

rank(score_coeff)

rank(c

ite_c

oeff

)

LASSO Model Findings: Determinants of Impact

• Terms of significance in ref reports include:– (+) “debate”– (+) “government”– (+) “from [years] to [years]”– (-) “counterfactual”– (+) “wage”

Could identify types of impact valued by different individuals & different fields

Basic Model

Journals/Editors face separable paper payoff function:

over accuracy a, impact i, fit f :

assuming fit is ex ante observable…

fiafiap ),,(

1,0,01,1 f i a

Basic Model

Journals/Editors maximize payoff:

by selecting a threshold based on available space and making decisions D accordingly:

𝐸൫Πpȁ�𝑠𝑎,𝑠𝑖൯= ඵ Πpሺ𝑎,𝑖,𝑓ሻ 𝐴ሺ𝑠𝑎,𝑎ሻ 𝐼ሺ𝑠𝑖,𝑖ሻ𝑎,𝑖 𝑑𝑎 𝑑𝑖

𝐷= ቊ𝑎𝑐𝑐𝑒𝑝𝑡 𝑖𝑓 𝐸൫Πp൯≥ 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑𝑟𝑒𝑗𝑒𝑐𝑡 𝑖𝑓 𝐸൫Πp൯≥ 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑

Methods

• Econometric analysis to understand form of payoff function Pp

• Dig deeper to develop metrics for a, i, f :– Measure submissions’ similarities to past accepted

articles (fit within journal niche)– Develop estimator of submission impact based on

referee language (potential impact)– Referee scores seem good proxy for accuracy– Also, count mentions of journal editors names in

submissions

Findings: Editor & Referee Preferences (ordered probits)

• Editors seem to show preference for:– (+) referee score– (+) fit of submission with journal niche– (+) potential impact– (++) interaction between potential impact & referee

score

• Referee scores show:– (+) very small taste for potential impact– no significant effect of fit with niche

• Terms of significance in ref reports (debate, from to years, etc.)

More Findings

• Evidence that published papers closer to journal’s niche get more impact/citations

• Papers mentioning an editor in first version more likely to get eventually accepted

• Editors are harder on papers that mention them• Evidence that editors differentiate/select higher

impact among papers that mention them (some do, others don’t; can score editors by this)

• Some editors choose harder referees (or are less likely to desk reject lower quality papers)

Summary

• Support Laband’s hypotheses that there is a great deal of rich information content in referee reviews

• Appearance of favoritism among editors may reflect better selection (as in Laband & Piette)

• Evidence of an interaction between accuracy and potential for citation impact that suggests it may be much more risky to produce and submit papers that are expected to receive high numbers of citations at certain journals.

• Characterizes budget constraints on referee capital (binding)• Characterize (low) publication space limits on high impact

publications

Insights for Development of Scientific Commons on Journals

• Referee reviews:– include full texts if possible– alternatively include check boxes for referees to indicate different types of

merit relevant to each journal• unique data sources or experimental setups• contributions to ongoing debates or policy

– computational methods for relevant types of merit by journal

• Anonymized Referee Scores• Measures of textual similarity to past accepted work• Anonymous flags on citations of referees or editors’ own work in a

paper– can be used to score referees and editors if desired

Simple to automatically compute given modern software technologies

Relevant to many types of scientific review (eg. society elections, grant awards, tenure decisions) and other processes where text accompanies data on preferences (employee promotion, appraisals, product reviews, health evaluations)

Insights for Development of Scientific Commons on Journals

• Balance between breadth of access and data detail, eg.:– referee/author identities,– review/manuscript texts– ability to connect datasets is critical!!!!

• Text-based metrics can be powerful:– LASSO for model selection on text– Recognition of editor names in first, last versions– Textual similarities (to past publications/submissions)– Automatically computable given modern software technologies

• Relevant to many types of scientific review – society elections– grant awards– tenure decisions– other processes where text accompanies data on preferences

(employee promotion, legal filings, appraisals, product reviews, health evaluations)

DISCUSSION/QUESTIONS

Ayeh Bandeh-Ahmadiabandeh@umd.edu