When Certainty is Uncertain
A presentation of “Linguistic Certainty, Task Certainty and Academic Success,” a research paper written for Cleveland State University’s
COM 633.
Julie A Cajigas
Rationale
Why CATA? Short et al. (2010) Validity Piece How has LIWC been validated? Is that enough?
What is Certainty?
Certainty has been examined throughout the Communication, Social Science and Mathematical literatures.
Types of uncertainty/certainty: Relational Uncertainty/Certainty Task Uncertainty/Certainty Cognitive Certainty Mathematical Certainty Linguistic Certainty?
Certainty
In examining COM 101 Journals, we chose Task Certainty. Task uncertainty refers to the difficulty and
variability of the work undertaken by an organizational unit. (Van de Ven, 1976)
Rationale for choosing task certainty over relational or other types of certainty.
Van de Ven’s Task Certainty Scale
Certainty
Certainty: The ability to predict outcomes in a given context.
What do we think of this definition?
Certainty in Academic Writing
Previously used in several research studies: Hyland and Milton (1997) Kennison (2003) Rubin (2010) Schmader et al. (2010) Carroll (2007)
Also looked at LIWC Linguistic Variables
Certainty and LIWC
Kennison 2003 Found that LIWC certainty had a negative relationship with
the subscales of the California Critical Thinking Skills Test. Carroll 2007
Saw an insignificant decrease in LIWC certainty over the course of an academic semester in student writing.
This begs the question: What is LIWC measuring exactly?
LIWC Dictionary
LIWC Dictionary
LIWC Certainty
Words included in the Pennebaker internal dictionary for certainty are included below.
Unlike other categories, which contain from 6-615 words in LIWC 2001, the certainty variable contains relatively few words at 30, though it is nearly impossible to think of words that should be added to this list, giving the dictionary good face validity. Certainty words account for an average of 1.1% (SD .9) of words across 43 studies as reported in the LIWC 2001 manual (Pennebaker et al., 2001).
For the purposes of the present study, it is most telling to look at the average for control writing, which is conceptually the closest to the student’s journal writing. The average for control writing over the 43 studies analyzed was .70. The mean for the student journals here is 1.25% (SD = .96), which is very close to the grand mean for certainty words, and close to the mean for emotional writing, which is 1.4%.
LIWC related Hypotheses/Research
Questions H1 Student task certainty for an assignment will
have a positive relationship with written certainty, as measured in that assignment.
H2 Student task certainty for an assignment will have a negative relationship with written tentativeness and discrepancy, as measured in that assignment.
Research Questions 1-2
Because of the following points from the literature, research questions emerged.
Certainty has been shown to decrease with familiarity, critical thinking and language proficiency, which is counterintuitive, and there are a relatively small number of studies with this effect (Hyland & Milton, 1997; Kennison, 2003; Carroll, 2007)
RQ1a-b How is written/linguistic certainty (as measured by LIWC) related to a. assignment grade expectations and b. course grade expectations?
RQ2 How is written/linguistic certainty related to the actual assignment grades?
Research Question 3
Several studies examine Tentativeness, Discrepancy and Certainty as measured by LIWC. Some add them together under CogMech and some use the individually. These seem like concepts which might have an inverse relationship.
RQ3 How are linguistic tentativeness and discrepancy related to linguistic certainty?
Research Question 4
Carroll (2007) showed that Linguistic Variables including Word Count, Words Per Sentence, Big Words etc., had a relationship with Student Achievement.
RQ4a-c How are student scores on LIWC linguistic variables related to student a. assignment grade expectations, b. course grade expectations and c. actual assignment grade?
Research Question 5
Van De Venn (1976) utilizes task certainty to predict task outcomes in organizational settings. Does Task Certainty relate to academic outcomes?
RQ5a-b How is student task certainty related to a. assignment grade expectations, b. course grade expectations and c. actual assignment grade?
Variables
Written/Linguistic Certainty Conceptually: The use of sure language in written
communication. Operationally: The LIWC Measure of Certainty
Task Certainty Conceptually: Task uncertainty refers to the difficulty and
variability of the work undertaken by an organizational unit. (Van de Ven, 1976)
Operationally: Van De Ven’s (1976) Task Uncertainty Index (utilized measures 1, 4, 5; with Cronbach’s Alpha of .71)
Academic Outcomes Conceptually: The outcome of an academic effort. Operationally: The student’s assignment grades, predicted
and actual, the student’s predicted course grade.
Sampling (The Easy Way)
Population: COM 101 Sections at Cleveland State University during Fall 2010.
Sampling Method: Census of Students in above population.
Sampling (Actual)
Actual Population: College students in courses which require academic writing participation around the nation.
Actual Sampling Method: a convenience sample of COM 101 students at Cleveland State University during the Fall semester 2010. 47 students completed the questionnaire and survey responses.
The unit of measurement for the content analysis was the journal, while the unit for the overall study was the student.
Sampling (Ideal)
Population: Students in colleges and universities nationally.
Sampling Method: Systematic Random Sampling Generate a list of all active college & university
email addresses, randomize them with a random number generator, and select them at intervals.
Procedure
Students in COM 101 were asked to submit an electronic copy of their week 9 journals, and to complete a task certainty instrument, which included the Van De Ven Task Uncertainty Index, two of Ashill& Jobber’s measures of Environmental Certainty, and a number of questions about the academic outcomes. 47 students completed the survey and turned in their journals.
Journals were cleaned up for spelling errors and run through the LIWC2001 CATA content analysis program.
Methods
Van De Ven’s (1976) measure of Task Uncertainty was coded and reverse coded so that certainty would be represented by a higher score. Measures from this scale were converted into Z scores and measured for scale reliability. The combination of questions with the highest reliability (Surety of finding info, surety of outcome and time it takes to be sure on the outcome) with a cronbach’s alpha of .71, were summed and became the new variable “TaskCert”
Methods (cont)
Q9, which is the student’s predicted grade is used in the analysis, along with Q17 which is the student’s predicted semester grade.
Student grades on these journals have not yet been received, and so hypotheses and RQ’s that require them are left out of the following results section.
Results: H1
H1 Student task certainty for an assignment will have a positive relationship with written certainty, as measured in that assignment. NOT SUPPORTED
Results H2
H2 Student task certainty for an assignment will have a negative relationship with written tentativeness and discrepancy, as measured in that assignment. NOT SUPPORTED
RQ 1
RQ1a-b How is written/linguistic certainty (as measured by LIWC) related to a. assignment grade expectations and b. course grade expectations? (No Significant Relationships)
RQ3
RQ3 How are linguistic tentativeness and discrepancy related to linguistic certainty? (No Significant Relationship)
RQ 4
RQ4a-c How are student scores on LIWC linguistic variables related to student a. assignment grade expectations, b. course grade expectations and c. actual assignment grade?
RQ 5
RQ5a-b How is student task certainty related to a. assignment grade expectations, b. course grade expectations and c. actual assignment grade?
Validation
This study is an attempt to validate a current measure (LIWC certainty) using criterion validity or predictive validity: Short et al. (2010) define predictive validity as “the extent to which the operationalization predicts other constructs consistent with theoretically derived a priori expectations.” Neuendorf (2002, 115) defines criterion validity as “the extent to which a measure taps an established standard or important behavior that is external to the measure.”
External Validity
The current study, with an N of 47 gathered via non-random sampling method has a low external validity. We should not attempt to generalize this study to a
larger population, but simply to repeat the study with funding and time allowing in a more externally valid way.
Face Validity
The face validity of this study is high, but has some challenges. Face validity is high because “on the face of things”
both the task uncertainty measure and the certainty measure in LIWC seem to be measuring the same thing.
One threat to face validity is the knowledge that LIWC was validated using emotional narratives rather than academic papers. This could reduce the validity of our overall argument.
Content Validity
“The extent to which the measure reflects the full domain of the concept being measured.” While task certainty is a validated measure, LIWC
certainty may only measure a small part of what we would consider “certainty.” While task certainty is very specific in what it applies to, LIWC certainty is not.
Construct Validity
As we saw in the results, Task Certainty has strong Construct Validity, with significant relationships to the student’s predicted grade, whereas LIWC certainty has little construct validity based on several of the research questions.
Limitations
Lack of true conceptual definition for LIWC Certainty. Idea for future study of LIWC Certainty
Small number of respondents/sampling technique. Limited variance in certainty measures because it
was the end of the semester, task has been repeated many times.
Additional limitations?
Discussion
Still working on this part – let’s discuss!
FinThank you all for your help and friendship this semester!
