Transparency: The Emerging Revolution in Qualitative Analysis

Andrew Moravcsik

Princeton University

APSA – DA-RT Workshop (September 2014)

Transparency: The Emerging Revolution in Qualitative Analysis

1. Transparency brings scholars together.

2. Scholars have been thinking about what transparency means for qualitative researchers.

3. “Active citation” (digitally enabled citation) is emerging as the lead standard for qualitative transparency.

4. AC provides transparency cost-effectively.

5. Much has been done. Now journals are getting involved.

Transparency as an Ideal

“The idea is to try to give all the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.”

-- Richard P. Feynman

Transparency as an Ideal

“History has this in common with every other science: that the historian is not allowed to claim any single piece of knowledge, except where he can justify his claim by exhibiting… the grounds upon which it is based [and] what the evidence at his disposal proves about certain events.”

-- R. G. Collingwood

What is History?

What is Research Transparency?

For evidence-based knowledge claims, researchers should achieve research transparency:

• Data Transparency: Provide access to data.

• Analytic Transparency: Explicate the link from data to descriptive/causal conclusions.

• Process Transparency: Show procedures used to collect, generate or choose the data, theory, and methods.

What does research transparency mean for qualitative research?

• Ideal-typical qualitative research

•Few cases (not many)

•Process observations

(not dataset observations)

•Textual evidence (not quantitative)

ACTIVE CITATION: “Digitally enabled citations” linked to annotated source excerpts in an appendix

TRADITIONAL CITATION

QUALITATIVE DATA ARCHIVING

HYPERLINKS TO ON-LINE SOURCES

DATABASES

Approaches to Transparency

MAIN TEXTContestable

Knowledge-basedClaim

CITATIONFootnote,

Endnote or In-Text

TRANSPARENCYAPPENDIX ENTRY

1. Citation2. Source Excerpt

3. Annotation[ 4. Optional Scan or Link to Full Source]

WHAT IS AN ACTIVE CITATION? BODY OF ARTICLE TRANSPARENCY APPENDIX (Remains Unchanged) (New)

Issues: data selection (“cherry picking”), robustness to theory choice and specification, precise methodological issues, relative weight of evidence.

• The main text, as usual.

• A special “methodological” appendix entry at the beginning of the transparency appendix:.

• Additions to individual annotations: e.g. the relative weight of evidence.

WHAT IS AN ACTIVE CITATION?Process Transparency

Tommy Lee Jones Rep. Thaddeus Stevens on the

Thirteenth Amendment

“The greatest measure of the nineteenth century was passed by corruption, aided and abetted by the purest man in America.”

…according to Steven Spielberg, who got it from historian Fawn Brodie (1959), who got it from popular lecturer James Scovel (1898)…

In Steven Spielberg’s recent film Lincoln, the abolitionist Representative Thaddeus Stevens (played by Tommy Lee Jones) returns home after the House of Representatives passed the Thirteenth Amendment (banning slavery). He utters the following striking phrase to his mulatto common-law wife, Lydia Smith, referring to Lincoln’s role in the passage of the amendment: “The greatest measure of the nineteenth century [was] passed by corruption, aided and abetted by the purest man in America.” The accuracy of this quotation has been questioned. One account, written much later, suggests that Stevens said this. Yet, if he did, it seems highly unlikely that he did so at home to his companion. FN: James Scovel, “Thaddeus Stevens,” Lippincott’s Monthly Magazine (April 1898), pp. 548-550.

An Example of Active Citation The Main Text and Footnote

AN EXAMPLEPart 1 of 4: Full Citation

James Scovel, “Thaddeus Stevens,” Lippincott’s Monthly Magazine (April 1898), pp. 548-550.

AN EXAMPLE Part 2 of 4: Textual ExcerptTo the writer of this sketch Mr. Stevens told the story of the legislation which gave to the black man his right to vote: … [548] His favorite amusement was…to spend the evening at Hall and Pemberton's Faro Bank…and over canvas-back and Veuve Clicquot champagne woo unmolested the goddess of fortune.…Stevens was never a heavy player, although I have seen him win fourteen hundred dollars on a twenty-dollar gold-piece as his only stake… [549] Influence from the White House secured votes against a favorite measure of Mr. Stevens for an air-line railway from Washington to New York, and…these same votes helped Mr. Lincoln's great amendment for emancipation. Of this legislative bargain Stevens said, ‘The greatest measure of the nineteenth century was passed by corruption, aided and abetted by the purest man in America.’ During the last thirty years of his life its unwritten romance was the unselfish and tender devotion with which Stevens was attended by Lydia Smith, a mulatto, who in her youth had great beauty of person. [550]

AN EXAMPLE Part 3 of 4: Annotation

The first published reference to Steven’s “corruption” statement was in 1898, by James M. Scovel, a retired New Jersey politician and lawyer. He quotes Stevens and links the reference to corruption to Lincoln’s willingness to secure votes for the Thirteenth Amendment by trading votes against a direct NY to Washington railway. Is he reliable on this point? On the positive side, Scovel’s account contains specific details, including a story Stevens allegedly told him about the passage of the Fifteenth Amendment and an anecdote about witnessing Stevens gamble. We know they Scovel and Stevens were political allies, and Stevens encouraged Scovel to run for Congress. A modest correspondence has been preserved. On the negative side, Scovel was hardly in Washington during the Lincoln Administration, and he did not write anything down for 33 years. In any case, Scovel’s account does not imply any connection between the quotation and Lydia Smith, Stevens’ common-law wife, though she is mentioned. Nor is it clear how anyone except Smith herself, who left no memoirs, would know if Stevens had said it to her alone.

Three Dimensions of Research Transparency

A transparent decision about what Stevens said rests on:

1. Data Transparency The Text

2. Analytic Transparency

How we interpret the text

3. Procedural Transparency

How we select the text and analytic methods

AN EXAMPLEPart 4 of 4: Scan or Link (Optional)

Transparency: What do scholars gain? Readers can appreciate the richness and rigor of qualitative data

and methods.

Scholars can display excellence, while journal editors, graduate advisors, departments and professional associations can better distinguish and promote it.

Scholars have an incentive to improve qualitative methods.

Colleagues can critique and debate empirical analysis.

Scholars (qualitative and quantitative) can reuse textual evidence for scholarly or public policy purposes.

Political scientists enjoy more opportunities for interdisciplinary cooperation with history, law, as well as public policy and journalism.

Some political scientists already do much of this.

AC only applies to sources documenting “contestable knowledge-based empirical claims.”

Precise length of quotations and annotations is discretionary.

Modern technology and advance planning lighten the load.

Writers, editors and publishers maintain existing formats.

Standard practice in other fields (law, history, classics).

Is Active Citation Too Costly?

Best Practices:Political Science

ACTIVE CITATION

TRADITIONAL CITATION

QUALITATIVE DATA ARCHIVING

HYPERLINKS TO ON-LINE SOURCES

DATABASES

Approaches to Transparency

Active Citation: What has been done?

Established a Qualitative Data Repository (QDR)

Secured NSF Funding

Sponsored Articles/Workshops on Conceptual and Practical (Human Subject, IP, and Logistical) Issues

Helped establish APSA Guidelines (with Quantitative and other communities)

Created NSF/QDR On-line Demonstration Portal: 20+ scholars retrofitting “classic” and forthcoming research

Organized Training Sessions at Qualitative/Multi-Method Institute and universities

Launched Interdisciplinary Outreach (SSRC, BITSS)

Reached Out to Journals: Special Issues and Presentations

Designed New Software

ACE Word Add-in

Active Citation…

Translate… Look Up

Active Citation…

Translate… Look Up

OUTPUT:- Word Document- Web Document

- Database (Access, Atlas…)

Active Citation: The Future Agenda

Work with journals on adoption, format, standards, and processes.

Convene interdisciplinary conversations and learn about common problems and “best practices” (SSRC)

Develop better guidance and classroom materials for researchers, teachers and policy analysts

Work with software providers (e.g. database software)

Reach out to journalists, policy-makers, government agencies and NGOs

Not just political science…

Not just the sciences in general…

The world of information is going this way.

Why is active citation inevitable?

Best Practices:Natural Science

Example: When did Polar Bears Evolve?

Article

Supplementary Materials

Best Practices: Legal Academia(Yale On-Line Law Review)

Best Practices: Policy Analysis

Best Practices: Specialized Journalism

Best Practices: Classified Intelligence Reports

Transparency: The Emerging Revolution in Qualitative Analysis

Andrew Moravcsik

Princeton University

APSA – DA-RT Workshop (September 2014)

Transparency: The Coming Revolution in Qualitative Analysis

Andrew Moravcsik

Princeton University

IQMR – Syracuse University (June 2014)

Transparency: Traditional Footnotes vs.Active Citations

  Traditional Citation Active Citation

Data Transparency

De facto costs vary from moderate to prohibitive.

Mandatory excerpt provides “one click” basic transparency.

Optional scan/link creates option for maximum within IP, human

subject and logistical constraints.

Analytical Transparency

Rarely possible, increasingly excluded by

current formats. Mandatory.

Production Transparency

Only within text of article.Special section of transparency

appendix plus any dedicated entries.