11

Problems, and little Problems, and little victories, in transcript victories, in transcript

analysisanalysis

Patrick J. FahyPatrick J. Fahy

Athabasca University, Centre for Distance Athabasca University, Centre for Distance EducationEducation

CIDER PresentationCIDER Presentation30 November 200730 November 2007

22

OutlineOutline

1.1. Why TA?Why TA?

2.2. Validity issuesValidity issues

3.3. Ethics principlesEthics principles

4.4. Implications for practiceImplications for practice

33

Why transcript analysis?Why transcript analysis?

• A conference transcript is “a gold mine of A conference transcript is “a gold mine of information concerning the psycho-social information concerning the psycho-social dynamics” among the participants dynamics” among the participants (Henri, (Henri,

1992,1992, p. 118)p. 118). . • CMC is related both to achievement and CMC is related both to achievement and

attitudes attitudes (Fulford & Zhang, 1993).(Fulford & Zhang, 1993).

• CMC is a way to establish CMC is a way to establish instructorinstructor and and social presence – social presence – facilitators of facilitators of online online community community (Rourke, Anderson, Garrison, & Archer, 1999)(Rourke, Anderson, Garrison, & Archer, 1999)..

– Community is “important to the success of” online learners Community is “important to the success of” online learners (Conrad, 2005).(Conrad, 2005).

44

Why TA? Why TA? (con’t)(con’t)

• CMC is common,CMC is common, but how to manage but how to manage CMC, and how CMC-based interaction CMC, and how CMC-based interaction improves learning and online improves learning and online experience, is not well understoodexperience, is not well understood..– Community constructed - somehow - Community constructed - somehow -

through language, in CMCthrough language, in CMC (Vrasidas & (Vrasidas &

McIsaac,1999)McIsaac,1999).. • ““Further study is needed to describe the discourse Further study is needed to describe the discourse

devices used in CMC to establish community” devices used in CMC to establish community” (Lapadat, (Lapadat, 2007, p. 62)2007, p. 62)

55

Validity - issuesValidity - issues

Lack of discriminant capabilityLack of discriminant capability– Many postings in few categories Many postings in few categories

(Gunawardena, Lowe, & Anderson, 1997; Kanuka & (Gunawardena, Lowe, & Anderson, 1997; Kanuka &

Anderson, 1998)Anderson, 1998)..– Postings could be coded into more Postings could be coded into more

than one category; than one category; category category ambiguity ambiguity (Zhu, 1996).(Zhu, 1996).

66

Validity – more issuesValidity – more issues

Instruments too complex, too many Instruments too complex, too many codescodes– Gunawardena et al. (1997): 20 Gunawardena et al. (1997): 20

categories (5 “phases”)categories (5 “phases”)– Cookson & Chang, 1995: 16 codesCookson & Chang, 1995: 16 codes– Rourke et al., 1999: 12 indicatorsRourke et al., 1999: 12 indicators– Higgins (1998): 20Higgins (1998): 20– Zhu (1996): 8Zhu (1996): 8

77

UnreliabilityUnreliability

– inter-coder reliability is lowinter-coder reliability is low•Sometimes, reliability data not Sometimes, reliability data not

reportedreported

88

Questionable solutionsQuestionable solutions

– Collaborative Collaborative (group) (group) codingcoding (engineered “coder drift”).(engineered “coder drift”).

– Mystical terminologyMystical terminology•““units of meaning,” “discourse units,” units of meaning,” “discourse units,”

“message units,” “message units,” etc.etc.

– Coding of whole messages into 1 Coding of whole messages into 1 categorycategory•““One code fits all”One code fits all”

99

Considerations regarding Considerations regarding validity validity

1.1. Principle of Principle of over-determination over-determination

2.2. Principle of Principle of commensurate commensurate complexitycomplexity

3.3. Dangers of Dangers of Type 1 errorsType 1 errors

1010

Principle of Principle of over-over-determinationdetermination

Social events usually have more causes Social events usually have more causes than are strictly required for them to than are strictly required for them to occur.occur.

• Contrary to Contrary to Occam’s Razor, Occam’s Razor, simplest explanations simplest explanations maymay not be truestnot be truest– ““Seek simplicity, then distrust it.” (Whitehead)Seek simplicity, then distrust it.” (Whitehead)

• ““... human development is the result of many ... human development is the result of many overlapping spheres of influence....” (Gibson, 2000) overlapping spheres of influence....” (Gibson, 2000)

• ““Unlike rare diseases such as Huntington’s, where a Unlike rare diseases such as Huntington’s, where a single genetic variation guarantees that a carrier will single genetic variation guarantees that a carrier will be afflicted, be afflicted, common diseases are triggered by a common diseases are triggered by a complex array of factorscomplex array of factors, including multiple genes , including multiple genes each exerting a modest effect.” each exerting a modest effect.” [Genes for several common [Genes for several common diseases, (2007, September/October). diseases, (2007, September/October).

1111

Principle ofPrinciple of commensurate commensurate complexitycomplexity

– Social principles cannot usually be Social principles cannot usually be at once at once general, accurategeneral, accurate, and , and simplesimple. . [Thorngate, W. (1976).][Thorngate, W. (1976).]

1212

A special problemA special problem

Type 1 error:Type 1 error:• rejection of a null hypothesis that is, in rejection of a null hypothesis that is, in

fact, fact, truetrue..

1313

EthicsEthics

• In Canada, the In Canada, the Tri-Council Policy Tri-Council Policy Statement: Ethical Conduct for Research Statement: Ethical Conduct for Research Involving Humans Involving Humans (TCPS, 2003), applies (TCPS, 2003), applies to all research, funded or not. to all research, funded or not. – Developed and jointly issued by the Medical Developed and jointly issued by the Medical

Research Council of Canada (MRC), the Research Council of Canada (MRC), the Natural Sciences and Engineering Research Natural Sciences and Engineering Research Council of Canada (NSERCC), and the Social Council of Canada (NSERCC), and the Social Sciences and Humanities Research Council of Sciences and Humanities Research Council of Canada (SSHRCC). Canada (SSHRCC).

1414

ProblemsProblems

• Research ethics boards (REBs) and Research ethics boards (REBs) and researchers disagree on what is “ethical.”researchers disagree on what is “ethical.”

• ... “an underlying current of distrust ... ... “an underlying current of distrust ... exists within the [Humanities] community exists within the [Humanities] community vis-à-vis the TCSP” vis-à-vis the TCSP” (Owen, Robert, Burgess, Golfman & (Owen, Robert, Burgess, Golfman &

Sykes, 2001, p. 4).Sykes, 2001, p. 4). – ““Review boards pose threat to social scientists’ Review boards pose threat to social scientists’

work.” work.” ((The Wall Street Journal, The Wall Street Journal, 2004)2004)

1515

Where is the Where is the real real problem?problem?

• Ethics issues Ethics issues in clinical/biomedical in clinical/biomedical researchresearch– Tuskegee syphilis study (1932 – 1972) Tuskegee syphilis study (1932 – 1972) – Polio vaccine studies on children (1950s)Polio vaccine studies on children (1950s)– Willowbrook hepatitis studies (1963 – Willowbrook hepatitis studies (1963 –

1966) 1966) – Drug trials (Olivieri vs. Apotex, 1998)Drug trials (Olivieri vs. Apotex, 1998)– Others?Others?

1616

Key ethical principles:Key ethical principles:Getting minimal risk research Getting minimal risk research approvedapproved

• EthicsEthics: from : from ethos = “custom.” ethos = “custom.” – TCPS: REBs to apply ethical principles “in TCPS: REBs to apply ethical principles “in

the context of the nature of the research the context of the nature of the research and of the ethical norms and practices of and of the ethical norms and practices of the relevant research discipline” the relevant research discipline” (TCPS, p. (TCPS, p.

i.9)i.9)..

• Emphasize maximization of benefits, Emphasize maximization of benefits, minimization of risks, potential harms.minimization of risks, potential harms.

1717

Respect for human dignityRespect for human dignity

• Persons are ends, not means.Persons are ends, not means.– Individual rights Individual rights normally normally take take

precedence over the advancement of precedence over the advancement of knowledgeknowledge

• Informed consent assures respect for Informed consent assures respect for individual rights... but collective individual rights... but collective rights are recognized, too.rights are recognized, too.

1818

Justice, inclusivenessJustice, inclusiveness

• No group should bear an unfair No group should bear an unfair burden in, or enjoy burden in, or enjoy disproportionately the benefits of, disproportionately the benefits of, research. research.

1919

Understand Understand informed informed consent consent ((The “Gold” standard The “Gold” standard ))Conditions under which informed consent Conditions under which informed consent

may be waived may be waived (TCPS, p. 2.1)(TCPS, p. 2.1)::1.1. The research is minimal risk.The research is minimal risk.2.2. The waiver is unlikely to adversely affect the The waiver is unlikely to adversely affect the

rights and welfare of the subjects.rights and welfare of the subjects.3.3. The research could not practicably be carried The research could not practicably be carried

out without it.out without it.4.4. Subjects will be provided with additional Subjects will be provided with additional

information after participation, “whenever information after participation, “whenever possible and appropriate.” possible and appropriate.”

5.5. A therapeutic intervention is not involved A therapeutic intervention is not involved [normally precluded by point 1].[normally precluded by point 1].

2020

RationaleRationale

• Majorities have rights:Majorities have rights:– Individuals may refuse to participate, and Individuals may refuse to participate, and

subjects who express concerns about a subjects who express concerns about a study may have the option of removing study may have the option of removing their data from the project; their data from the project; however:however: •This approach should be used This approach should be used onlyonly when the when the

elimination of the subject’s data will not elimination of the subject’s data will not compromise the validity of the research compromise the validity of the research design, and hence diminish the ethical value design, and hence diminish the ethical value of participation by other subjectsof participation by other subjects. . (TCPS, p. 2.3; (TCPS, p. 2.3;

emphasis added)emphasis added)

2121

• So long as the methodology is sound, all So long as the methodology is sound, all contributions are anonymous, contributions are anonymous, and identities and identities are protected in any reportsare protected in any reports, it may well , it may well be a greater ethical violation, in relation to be a greater ethical violation, in relation to participation, to deny the rights of the willing participation, to deny the rights of the willing majority than it would be to accede to the majority than it would be to accede to the wishes of the reluctant minority by refusing to wishes of the reluctant minority by refusing to approve a research proposal. approve a research proposal. – ““Nothing Nothing in this Policy should be interpreted to in this Policy should be interpreted to

mean that research subjects have the right to veto mean that research subjects have the right to veto a project, though they do, of course, have the right a project, though they do, of course, have the right to refuse to cooperate with the researcher(s)” to refuse to cooperate with the researcher(s)” (TCPS, (TCPS,

p. 1.2; emphasis added)p. 1.2; emphasis added). .

2222

Other limitations to informed Other limitations to informed consentconsent

• Preserve the integrity of the Preserve the integrity of the research, by avoiding “colour[ing] research, by avoiding “colour[ing] the responses of the subjects and the responses of the subjects and thus invalidat[ing] the research” thus invalidat[ing] the research” (TCPS, (TCPS,

pp. 2.2 – 2.3)pp. 2.2 – 2.3)..

2323

Individuals and groups not Individuals and groups not entitled to informed consent entitled to informed consent (Remember: this is the “gold standard” in the ethical treatment of human (Remember: this is the “gold standard” in the ethical treatment of human subjects)subjects)

• Public bodies, corporations, political Public bodies, corporations, political partiesparties

• Private organizationsPrivate organizations– ““... may not interfere with or veto research about their own ... may not interfere with or veto research about their own

conduct, their participation in any research about their conduct, their participation in any research about their behavior is not required, and they need not be approached for behavior is not required, and they need not be approached for their consent prior to research being conducted.” their consent prior to research being conducted.” (TCPS, p. 2-2)(TCPS, p. 2-2)

• Public figuresPublic figures– (“Certain types of research … may legitimately have a (“Certain types of research … may legitimately have a

negative effect on public figures ….” (TCPS, p. 1-6)negative effect on public figures ….” (TCPS, p. 1-6)

• Naturalistic observationsNaturalistic observations• Classroom, course evaluationsClassroom, course evaluations• Quality assurance, performance reviewsQuality assurance, performance reviews

2424

Secondary use of dataSecondary use of data

• ““Secondary use of data refers to the use in Secondary use of data refers to the use in research of data contained in records collected research of data contained in records collected for a purpose other than the research itself. for a purpose other than the research itself. Common examples are patient or school Common examples are patient or school records or biological specimens, originally records or biological specimens, originally produced for therapeutic or educational produced for therapeutic or educational purposes, but now proposed for use in research. purposes, but now proposed for use in research. This issue becomes of concern This issue becomes of concern onlyonly when data when data can be linked to individuals, and becomes can be linked to individuals, and becomes critical when the possibility exists that critical when the possibility exists that individuals can be identified in the published individuals can be identified in the published reportsreports.” .” ((TCPS, TCPS, p. 3.4; emphasis added)p. 3.4; emphasis added)

2525

The The realreal gold standard of gold standard of ethical research ethical research

ANONYMITY.ANONYMITY.

2626

Suggestions for practice Suggestions for practice ((praxispraxis))

1.1. Notice who speaks, who doesn’t.Notice who speaks, who doesn’t.• Invite the reluctant.Invite the reluctant.• Respect the desire for anonymity.Respect the desire for anonymity.

2.2. Notice hubs, spokes.Notice hubs, spokes.• Assure hubs are positive; check on the isolated Assure hubs are positive; check on the isolated

(people no one is talking or responding to).(people no one is talking or responding to).

3.3. Talk to everyone at least once Talk to everyone at least once • But don’t insist others do this.But don’t insist others do this.

4.4. Link, document.Link, document.• Be the literature/social link.Be the literature/social link.

2727

5.5. Be present.Be present.• ““80% of getting ahead is just showing 80% of getting ahead is just showing

up.” up.” (Woody Allen)(Woody Allen)

6.6. Give credit for participation.Give credit for participation.• What gets measured gets done.What gets measured gets done.

7.7. Set a standard.Set a standard.• What gets modeled gets done.What gets modeled gets done.

2828

CitationsCitations

• Cookson, Peter S. & Chang, Yu-bi. (1995). The Multidimensional Cookson, Peter S. & Chang, Yu-bi. (1995). The Multidimensional Audioconferencing Classification System (MACS). Audioconferencing Classification System (MACS). American Journal of American Journal of Distance Education, 9Distance Education, 9(3), 18-36.(3), 18-36.

• Fulford, C. P. & Zhang, S. (1993). Perception of interaction: The critical Fulford, C. P. & Zhang, S. (1993). Perception of interaction: The critical predictor in distance education. predictor in distance education. The American Journal of Distance The American Journal of Distance Education, 7Education, 7(3), pp. 8 – 21.(3), pp. 8 – 21.

• Genes for several common diseases. (2007, September/October). Genes for several common diseases. (2007, September/October). Technology review, 110Technology review, 110(5), p. 109.(5), p. 109.

• Gunawardena, C., Lowe, C. & Anderson, T. (1997). Analysis of a global on-Gunawardena, C., Lowe, C. & Anderson, T. (1997). Analysis of a global on-line debate and the development of an interaction analysis model for line debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17Journal of Educational Computing Research, 17(4), pp. 395-429. (4), pp. 395-429.

• Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), (Ed.), Collaborative learning through computer conferencing: The Najaden Collaborative learning through computer conferencing: The Najaden papers papers (pp. 117-136). Berlin: Springer-Verlag.(pp. 117-136). Berlin: Springer-Verlag.

• Medical Research Council of Canada. (2005). Tri-council policy statement: Medical Research Council of Canada. (2005). Tri-council policy statement: Ethical conduct for research involving humans. Ottawa: Government of Ethical conduct for research involving humans. Ottawa: Government of Canada. Available: Canada. Available: http://www.pre.ethics.gc.ca/english/policystatement/policystatement.cfm

2929

• Oshinsky, D. M. (2005). Oshinsky, D. M. (2005). Polio: An American story. Polio: An American story. Toronto: Oxford Toronto: Oxford University Press.University Press.

• Owen, M., Robert, L., Burgess, J., Golfman, N. & Sykes, S. (2001). Owen, M., Robert, L., Burgess, J., Golfman, N. & Sykes, S. (2001). Report Report to the Social Sciences and Humanities Research Council of Canada, to the Social Sciences and Humanities Research Council of Canada, Implementation of the Tri-Council Policy Statement on Ethics in Human Implementation of the Tri-Council Policy Statement on Ethics in Human Research (TCPS) (Humanities Project).Research (TCPS) (Humanities Project). Unpublished paper, Humanities and Unpublished paper, Humanities and Social Sciences Federation of Canada.Social Sciences Federation of Canada.

• Rourke, L., Anderson, T., Garrison, R. & Archer, W. (1999). Assessing Rourke, L., Anderson, T., Garrison, R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. social presence in asynchronous text-based computer conferencing. Journal of Distance Education, 14Journal of Distance Education, 14(2), pp. 50-71.(2), pp. 50-71.

• Thorngate, W. (1976). “In general” vs.. “it depends”: Some comments on Thorngate, W. (1976). “In general” vs.. “it depends”: Some comments on the Gergen-Schlenker debate. the Gergen-Schlenker debate. Personality and Social Psychology Bulletin Personality and Social Psychology Bulletin 2, 2, pp. 404 – 410.pp. 404 – 410.

• Vrasidas, C. & McIsaac, M. S. (1999). Factors influencing interaction in an Vrasidas, C. & McIsaac, M. S. (1999). Factors influencing interaction in an online course. online course. American Journal of Distance Education, 13American Journal of Distance Education, 13(3), pp. 22 – 36.(3), pp. 22 – 36.

• Zhu, E. (1996). Meaning negotiation, knowledge construction, and Zhu, E. (1996). Meaning negotiation, knowledge construction, and mentoring in a distance learning course. In mentoring in a distance learning course. In Proceedings of Selected Proceedings of Selected Research and Development Presentations at the 1996 National Convention Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology of the Association for Educational Communications and Technology (18th, (18th, Indianapolis, IN). Available from ERIC documents: ED397849. Indianapolis, IN). Available from ERIC documents: ED397849.

3030

Pat FahyPat Fahy

Athabasca University, Centre for Distance Athabasca University, Centre for Distance EducationEducation

patf@athabascau.ca

866-514-6234866-514-6234