Date post: | 21-Jul-2015 |
Category: |
Education |
Upload: | jeremy-donald |
View: | 237 times |
Download: | 1 times |
2008- 2009 Research Practices
Survey Analysis- an Assessment of
the impact of the QEP
Department of Institutional Research
Jessica Larsen, M.Ed.
Diane Saphire, Ph.D.
QEP Goals• As outlined in the QEP, there are five goals for
information literacy:
• Understand
• Access
• Use Ethically
• Evaluate
• Create
The QEP and Assessment
• QEP identifies tests, surveys, embedded questions, and rubrics as tools to use in assessing the progress on each of the stated goals.
• FYILLA- First Year Information Literacy in the Liberal Arts Assessment, now The Research Practices Survey is identified as one of these tools in the QEP.
• For more information on the history and development of the Research Practices Survey please see the assessment portion of the QEP
• Questions on The Research Practices survey address the first four goals
Research Practices Survey
• 34 Questions • Examines the information literacy
experiences, attitudes, approaches, and skills of first year students▫ Part test of skills and knowledge▫ Part survey of their beliefs and attitudes about
academic research
• QEP Data- 2008/2009▫ Administered in Fall of 2008▫ Re-Administered in Spring of 2009 108 students participated in both administrations.
• Pre-QEP Baseline Data- 2006/2007 academic year▫ Administered in Fall of 2006▫ Re-Administered in Spring of 2007 63 repeated subjects
Benchmark scores were computed by scoring relevant survey questions, summing the scores for each individual, and creating a percentage score, by dividing earned score by total possible score.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Fall'08 Spring '09
Gr
ad
e E
qu
iva
len
cy
Understand
Use Ethically
Access
Evaluate
•Understand the varieties of information sources availableStudents will demonstrate an understanding of the nature of informationsources, such as books, journals, newspapers, Websites, and media, andan understanding of how they vary in audience orientation and authority.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Fall Spring
Gr
ad
e E
qu
iva
len
cy
2006-2007 2008-2009
Percentages represent the proportion of the sample that responded that they had used the following sources
Dramatic decline in the use of encyclopedias, almanacs, and dictionaries.
The lack of an increase in 08-09 academic and research journals may be due to the increase in online resources
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Library BooksEncyclopedias, alamanacs, dictionariesAcademic or research journalsNewspapers or Magazines for General Public
Of
sa
mp
le
Fall '06 Spring '07 Fall '08 Spring '09
PRINT sources used for research during
past academic year?
ONLINE sources used for research in
the past academic year?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
General Search EnginesJournals, Magazines, Newspapers, or EncyclopediasLibrary Catalog Indexes or databases Google Scholar
Of
sa
mp
le
Fall '08 Spring '09 Fall '08 Spring '09
Identify source:
Jorgenson, Lars W. “Reinterpreting Navajo Rites.”
Navajo Culture 6 (1946): 469-78.
Question Prompt:
Indicate whether the item is an entire book, a journal
article, or a portion of a book
• In 08/09 here is a 4.8% increase in the number of students who answered correctly by the spring semester
• By the Spring in 2009, 67.6% of the respondents could not identify a journal article citation.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Fall '06 Spring '07 Fall '08 Spring '09
Co
rr
ec
tly
Id
en
tifi
ed
Correctly Identify
Sources from a CitationStudents were presented with three citations:
• An academic journal article,
• a book
• A portion of a book
•Question asked them to correctly identify the source
•The graph represents the percentage of the respondents that answered correctly.
•Differences were not significant
Citations:
Allen, Glover Morril. Bats. Cambridege: Harvard University Press, 1939.
Tanaka, Kazuko. “The New Feminist Movement in Japan, 1970-1990.” In Japanese Women, edited by KumikoFujimura-Fanselow. New York: Feminist Press, 1995.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Book Portion of a Book
An
sw
er
ed
Co
rr
ec
tly
Fall '06 Spring '07 Fall '08 Spring '09
Defining CitationIn a scholarly article or research paper, a citation is:
1. A direct quotation from someone else’s written work
2. Source information from any ideas or text from someone else’s written work (credited response)
3. The physical location of a source (book, journal, etc.)
4. All of the above
5. Don’t know
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
An
sw
er
ed
Co
rr
ec
tly
Fall '06 Spring '07 Fall '08 Spring '09
1. Primary sources are more scholarly than secondary sources.
2. Primary sources are old; secondary sources are new.
3. (Credited Response) Primary sources examine subjects first-hand; secondary sources examine the findings of other scholars.
4. Primary sources are more appropriate for academic projects than are secondary sources.
5. All of the above are equally accurate
6. Don’t know0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6
Of
sa
mp
le
Fall '06 Spring '07 Fall '08 Spring '09
Distinguish between a Primary and a Secondary Source
When two researchers disagree, one
must be wrong
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
strongly agree agree disagree strongly disagree
Of
Re
sp
on
da
nts
Fall '06 Spring '07 Fall '08 Spring '09
Good research yields clear results; poor research
yields ambiguous results
•Question asks: “To what extent to you agree or disagree with this statement.”
•Disagree / strongly disagree are the credited responses.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Dis
ag
re
e/S
tro
ng
ly D
isa
gr
ee
Fall '06 Spring '07 Fall '08 Spring '09
•Access information efficiently and effectivelyStudents will understand and apply techniques for accessing information which mayinclude general searching principles, accessing appropriate Web-basedresources, becoming familiar with specialized collections, and using Interlibrary Loan.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Fall Spring
Gr
ad
e E
qu
iva
len
cy
2006-2007 2008-2009
Application of Skills Which of the following is the correct way to truncate a search word?1. Typing in only the first
syllable of the word as a keyword
2. Combining search words with “and,” “or,” or “not”
3. Using * or ! Symbol in place of the last few letters of the word (credited)
4. None of the above5. Don’t know
Which of the following searches would retrieve the MOST results in an online search?1. Movies OR films
(credited)2. Movies AND films3. Movies NOT films4. Movies INSTEAD OF
films5. Don’t know
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Truncate Most Results
an
sw
er
ed
co
rr
ec
tly
Fall '06 Spring '07 Fall '08 Spring '09
You retrieve the following information
from a library catalog search:
Gothic modernisms / edited by Andrew Smith and Jeff Wallace.
New York, N.Y. : Palgrave, 2001.Stacks PR888.M63 G67 2001 AVAILABLExii, 232 p. ; 23 cm.Includes bibliographical references and index.English fiction -- 20th century -- History and criticism.Modernism (Literature) -- Great Britain.Modernism (Literature) -- United States.Gothic revival (Literature)Smith, Andrew, 1964-Wallace, Jeff, 1958-
Application of Skills Which of the following would be the MOST efficient way to find a comprehensive listing of other books in the catalog on this topic?
1. Identifying other books written by the same authors
2. Examining the bibliography in the book (Partial Credit)
3. Scanning the shelves where the book is located to see what books are nearby (Partial Credit)
4. Searching again using the subject headings that most closely match your research topic (Full Credited Response)
5. Don’t know.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Full Credit Partial Cred
An
sw
er
ed
Co
rr
ec
tly
Fall '06 Spring '07 Fall '08 Spring '09
Yield a list of relevant scholarly articlesWhich of the following is likely to yield the most comprehensive list of relevant scholarly articles for a research project?
1. Searching an electronic Index or database in a specific academic field (History, Biology, Music, etc.) (Credited Response)
2. Using a general Internet search like Google or Yahoo
3. Paging through print volumes of an academic journal in a specific academic field (Partial Credit)
4. Searching the library catalog
5. All of the above are equally effective
6. Don’t know
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Full Credit Partial Credit
Of
sa
mp
le
Fall '06 Spring '07 Fall '08 Spring '09
•Understand the concept of intellectual property and theeconomic, legal, and social contexts of information, and useinformation ethicallyStudents will understand the concepts of plagiarism and copyright and willappropriately use citation/documentation systems in their work. Students willdemonstrate an understanding of the University Academic Honor Code.
0%10%20%30%40%50%60%70%80%90%
100%
Fall Spring
Gr
ad
e E
qu
iva
len
cy
2006-2007 2008-2009
A citation is not required when?
• You are paraphrasing, rather than quoting a source
• More than one source says the same thing
• You are describing your own findings or analysis (correct response)
• All of the above
• Don’t know
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
An
sw
er
ed
co
rr
ec
tly
Fall '06 Spring '07 Fall '08 Spring '09
Evaluating information and its sources
0%
20%
40%
60%
80%
100%
Fall Spring
2006-2007 2008-2009
A peer-reviewed or refereed journal is BEST described as:
0% 50% 100%
Publishes reviews of other articles
Publishes articles approved by other scholars (credited …
Includes only articles written collaboratively by peers
Includes references for each article it publishes
Don't know
Fall '06 Spring '07 Fall '08 Spring '09
Researchers must distinguish between academic journals
and popular magazines. Which of the following
statements is LEAST ACCURATE?0% 20% 40% 60% 80% 100%
Academic journal articles provide objective facts (credited response)
Academic journal articles usually include a list of scholarly references
The intended audience for academic journals is mainly scholars
Editors & authors of acad. journal articles are usually employed in higher ed.
Don't know
Fall '06 Spring '07 Fall '08 Spring '09
Which of the following is the best way to determine
whether to use a particular source?
0% 20% 40% 60% 80% 100%
Whether it is a Print or internet source
How recently it was published
How easily you can get the source
Whether the source is scholarly
Whether the author is highly respected
All of the above are equally valid
Fall '06 Spring '07 Fall '08 Spring '09
Identify the least appropriate source
for a research paper0% 20% 40% 60% 80% 100%
University Press book
Encyclopedia Volumes
Book retrieved from electronic web archive
Journal article
Personal Webpage (credited response)
Fall '06 Spring '07 Fall '08 Spring '09
Library Use
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Once a week or more Once or twice a month
A few times a year Never
Of
sa
mp
le
Fall '06 Spring '07 Fall '08 Spring '09
• Prompt: How often in the past academic year did you use resources from a school library for school assignments, research projects or other academic purposes?
• Fall represents the students’ use of their high school library
• Spring represents the students’ use of their college library
Number of assignments requiring at
least 3 sources
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Five or more Three or four One or two None
Of
sa
mp
le
Fall '06 Spring '07 Fall '08 Spring '09
How many assignments, papers
, or research projects have you completed in the
past academic year that required you to include at least three
sources in a bibliography, References, or works cited
list?
Assignments requiring a specific
format (APA, MLA, Chicago, or some
other)• In the past academic
year, when you were given research project assignments, how often were you required to use a specific format (such as APA, MLA, Chicago, or some other style) for the sources in your bibliography
• Possible Responses: Almost always, Often, Sometimes, Rarely, Never/not applicable
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Fall '06 Spring '07
Fall '08 Spring '09
Which of the following best describes
the way you pace your work on a
research assignment0% 20% 40% 60% 80% 100%
I do most of the work soon after an assignment is given
I divide the work pretty equally across the available time
I do a little work soon after the assignment is given, but do most of
it toward the end
I do all of the work just before or on the due date
Fall '06 Spring '07 Fall '08 Spring '09
Seeking help or advice0% 20% 40% 60% 80% 100%
Teachers or Professors
Librarians
Parents/Other Adult Family
Friends, Classmates, or Siblings
Writing labs, centers, or help groups
Help screens, online tutorials or other
Fall '06 Spring '07 Fall '08 Spring '09
In the past academic
year, when you were working on
research assignments, how often did you
seek help or advice from each of the following?
% of students who answered
“almost always” or “often”
Consulting a Librarian
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Five or more times Three or four times Once or twice Never
Of
sa
mp
le
Fall '06 Spring '07 Fall '08 Spring '09
• Prompt: Over the course of the past academic year, how often did you talk with a librarian about a research project you were doing?
Conclusions:
• Students are making significant gains over the course of their freshman year.
• However, students are still demonstrating a lack of basic research skills and competencies.
• The Research Practices Survey may not be capturing student progress. Faculty in-course assessments may be the best method to collecting valid data that truly reflects student outcomes.
2008/2009 TU Repeat Participant
Gender Breakdown
Total
Freshman
Class
All Respondents Repeated
Fall Spring
Females 53.1% 64.6% 61.0% 67.6%
Males 46.9% 35.4% 39.0% 32.4%
n 655 161 292 108
2006-2007 TU Participant
Gender Breakdown
Total
Freshman
Class
All Respondents Repeated
Fall Spring
Females 53.9% 62.9% 69.8% 70.5%
Males 46.1% 37.1% 30.2% 29.5%
n 660 167 129 63