Exploring the RolesExploring the Rolesof Faculty Supervision: of Faculty Supervision:
Improving Qualitative Doctoral Improving Qualitative Doctoral Dissertation MethodologyDissertation Methodology
Dan Kaczynski, PhDDan Kaczynski, [email protected]@uwf.edu
Research ProblemResearch Problem
Increasing emphasison quality
Understand assessment practicesStrengthen qualitative research skills
Develop future researchers
Morequalitative
dissertations
Shiftingsupervisory
roles
Open DiscussionOpen Discussion
Do you use technology Do you use technology in your research?in your research?
Open DiscussionOpen Discussion
How should we explore the tensions How should we explore the tensions within and between:within and between:
Assessing QualityAssessing Quality Adoption of QDASAdoption of QDAS
Qualitative SoftwareQualitative SoftwareNVivo, MAXqda, Atlas ti, QDA Miner, Qualrus, TransanaNVivo, MAXqda, Atlas ti, QDA Miner, Qualrus, Transana
Kaczynski (2004) http://www.aare.edu.au/04pap/kac041065.pdf
What is Good Qualitative Research?What is Good Qualitative Research?
What does good work look like?What does good work look like?
Identify indicators of Identify indicators of
quality in a thesis:quality in a thesis:
Identify common errors Identify common errors in a thesis:in a thesis:
What is Quality?What is Quality?
The researchers logic of The researchers logic of justificationjustification
“ “Flaws in the logic of justification can potentially Flaws in the logic of justification can potentially occur anywhere in the inquiry process. The occur anywhere in the inquiry process. The nature of such flaws and where they occur can nature of such flaws and where they occur can jeopardize the soundness of a study in one or jeopardize the soundness of a study in one or more ways.”more ways.”
((Piantanida & GarmanPiantanida & Garman 1999, p. 147) 1999, p. 147)
Types of Quality CriteriaTypes of Quality Criteria
Philosophical Criteria Philosophical Criteria (Lincoln, 1995; Lincoln & Guba, 1985)(Lincoln, 1995; Lincoln & Guba, 1985)
Procedural CriteriaProcedural Criteria(Creswell, 1998)(Creswell, 1998)
Philosophical CriteriaPhilosophical Criteria (Lincoln & Guba, 1985)(Lincoln & Guba, 1985)
CredibilityCredibility Is the work authentic?Is the work authentic?
TransferabilityTransferability Will the work fit outside this situation?Will the work fit outside this situation?
DependabilityDependability Is the researcher consistent?Is the researcher consistent?
ConfirmabilityConfirmability Are interpretations defensible?Are interpretations defensible?
Procedural CriteriaProcedural Criteria
Quality of methodsQuality of methods(open-ended interviews)(open-ended interviews)
Quality of dataQuality of data(verbatim long transcripts)(verbatim long transcripts)
Quality of data analysisQuality of data analysis(comprehensive data treatment)(comprehensive data treatment)
(Silverman, 2004 [Sacks, 1984])(Silverman, 2004 [Sacks, 1984])
Standardized Procedural CriteriaStandardized Procedural Criteria(controversial checklists or guidelines)(controversial checklists or guidelines)
Does the title reflect the study focus?Does the title reflect the study focus? Is the problem socially important?Is the problem socially important? Is the literature review comprehensive?Is the literature review comprehensive? Has study conformed to ethics standards?Has study conformed to ethics standards? Are issues of sampling discussed?Are issues of sampling discussed? Did findings answer the questions?Did findings answer the questions? Was study written convincingly? Was study written convincingly? Are issues of trustworthiness addressed?Are issues of trustworthiness addressed?
Quantitative and Qualitative CriteriaQuantitative and Qualitative Criteriafor Assessing Research Quality and Rigorfor Assessing Research Quality and Rigor
Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002) p. 30Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002) p. 30
Quantitative Quantitative term term
Qualitative Qualitative termterm
Strategy employedStrategy employed
Internal validityInternal validity CredibilityCredibility Prolonged engagement in fieldProlonged engagement in fieldUse of peer debriefingUse of peer debriefingTriangulationTriangulationMember checksMember checksTime samplingTime sampling
External validityExternal validity TransferabilityTransferability Provide thick descriptionProvide thick descriptionPurposive samplingPurposive sampling
ReliabilityReliability DependabilityDependability Create an audit trailCreate an audit trailCode-recode strategyCode-recode strategyTriangulationTriangulationPeer examinationPeer examination
ObjectivityObjectivity ConfirmabilityConfirmability TriangulationTriangulationPractice reflexivityPractice reflexivity
Transparent AssessmentTransparent Assessment
Explore rich diversity of meaningsExplore rich diversity of meanings Sensitized appreciation of worthSensitized appreciation of worth Deeper assessment of analysisDeeper assessment of analysis Multiple paths to look inside Multiple paths to look inside Transparency strengthens credibilityTransparency strengthens credibility
Data CollectionData Collection
Stage 1Stage 1 Stage 2Stage 2 Stage 3Stage 3
Survey Survey ResearchResearch
Interviews/Interviews/ Documents Documents
Document Document ReviewReview
Supervisory Supervisory AbilityAbility
Quality Factors Quality Factors TechnologyTechnology
ResourcesResources
Professional Professional DevelopmentDevelopment
PerceptionsPerceptions
ProcessesProcesses
Assessment Assessment PracticesPractices
Findings: Knowledge,Findings: Knowledge,Ability, and ConfidenceAbility, and Confidence
Satisfactory Satisfactory or Higheror Higher
MM SDSD
SupervisingSupervising 73.9%73.9% 3.483.48 1.281.28
Serving on a Serving on a committeecommittee
91.3%91.3% 4.174.17 .94.94
Judging Judging qualityquality
91.3%91.3% 4.094.09 .95.95
Findings: TechnologyFindings: TechnologyTools Used in AssessmentTools Used in Assessment
0
2
4
6
8
10
12
14
NO NA AT HR IR MQ NV QM QU SS TR OT
Technology Tools
No
. of
Res
po
nd
ents
52% (None)30% (Not applicable)17% NVivo4% InfoRapid
Findings: ResourcesFindings: ResourcesConsulted for ExpertiseConsulted for Expertise
0
2
4
6
8
10
12
14
16
18
20
NA CF SP CE WA WC
Resources Used
Nu
mb
er o
f R
esp
on
den
ts83% Others70% Publications30% Conference
workshops22% Campus
workshops13% Continuing
education
Findings: ConceptualizationsFindings: Conceptualizationsof Quality (cont.)of Quality (cont.)
AIAI: Alternative interpretationsAlternative interpretationsCE: Consideration of ethical CE: Consideration of ethical issuesissuesAG: Ability to generalize findingsAG: Ability to generalize findingsHC: Hierarchical code structureHC: Hierarchical code structureMC: Member-checkingMC: Member-checkingMSMS: MemosMemosAT: Methodological audit trailAT: Methodological audit trailPD: PD: Peer debriefing Peer debriefing PF: Prolonged field engagementPF: Prolonged field engagementAS: Qualitative data analysis AS: Qualitative data analysis softwaresoftwareRORO: Researcher objectivityResearcher objectivitySSSS: Sampling strategiesSampling strategiesSDSD: Self-disclosureSelf-disclosureSC: Social context SC: Social context TOTO: Theoretical orientationTheoretical orientationTN: Triangulation TN: Triangulation VYVY: ValidityValidity
Stage 2 Findings: Critical NeedsStage 2 Findings: Critical Needs
Building knowledge and skillsBuilding knowledge and skills– Moving beyond superficial assessmentMoving beyond superficial assessment– Significance of researcher transparencySignificance of researcher transparency– Teaching students to self-assessTeaching students to self-assess
Building a community of practiceBuilding a community of practice– Strengthening qualitative research skillsStrengthening qualitative research skills– Sharing assessment strategiesSharing assessment strategies– Engaging in professional developmentEngaging in professional development
Stage 3 Findings:Stage 3 Findings:Role of TechnologyRole of Technology
0
10
20
30
40
50
60
70
80
90
Total Dissertations Analysis Software
(Not identified) Mixed Methods Qualitative Quantitative
Study FindingsStudy Findings
Highly favorable attitudes toward Highly favorable attitudes toward qualitative researchqualitative research
Diverse conceptualizations of qualityDiverse conceptualizations of quality Need for alternative assessment Need for alternative assessment
frameworksframeworks Need and desire to strengthen Need and desire to strengthen
knowledge and skillsknowledge and skills Need for professional developmentNeed for professional development
Future ChallengesFuture Challenges
Progressing research methodsProgressing research methods
Mixed Mixed →→ Blended Blended →→ Integrated Integrated
Emerging research innovationsEmerging research innovationsmainstream adoption of QDASmainstream adoption of QDAS
Positioning quality research Positioning quality research standardsstandards
Future Research QuestionsFuture Research Questions
What does it mean to disclose or conceal the role What does it mean to disclose or conceal the role of technology?of technology?
Does nondisclosure of analysis software imply the Does nondisclosure of analysis software imply the presumption that the use of technology is presumption that the use of technology is ubiquitous and commonly accepted?ubiquitous and commonly accepted?
Does nondisclosure of QDAS suggest a student’s Does nondisclosure of QDAS suggest a student’s fear of the supervisor’s acceptance or fear of the supervisor’s acceptance or sanctioning?sanctioning?
In what ways and under what conditions does a In what ways and under what conditions does a technological tool become a barrier to the technological tool become a barrier to the learning process for the teacher and the learner? learning process for the teacher and the learner?