A Standardized Approach to Assessment – Phase 2A Standardized Approach to Assessment – Phase 2

New Improvements Based on Student , Consultant & Library Faculty Feedback New Improvements Based on Student , Consultant & Library Faculty Feedback

Shannon M. StaleyEducation & Web Services LibrarianShannon M. StaleyEducation & Web Services Librarian

In 2008 / 2009, We Began By:

Developing and testing an assessment tool that could:

• Facilitate the development of pre and post achievement tests

• Link survey questions to learning outcomes and ACRL standards

• Deliver real time reports and statistical analysis

In 2010, We Asked the Question:How can we improve the tool and the assessment process?

1.Usability testing sessions with librarians who have not used the tool

2.More cognitive interviews with students to improve question development

3. Improved statistical analysis

4.Focus group with librarians who have used the tool

1. Usability Testing

Usability Sessions: Changes you asked for

Comment: I didn’t understand where I was when I entered the tool.

Usability Sessions: Changes you asked for

Remedy: A clearly defined menu as the starting screen

Usability Sessions: Changes you asked for

Comment: Make it easier to toggle from my personal account to all questions and possibly select and copy some for my account.

Usability Sessions: Changes you asked for

Remedy:• “View Other Librarians’ Questions” was changed to “View All

Questions.” • It’s easier to add and remove questions. • The “My Questions” view and “All Questions” view are different

colors for more distinction. • We have indicated which questions have been standardized.

Usability Sessions: Changes you asked for

Comment: I didn’t understand when the survey was actually created – also the link to the survey was on a site somewhere.

Usability Sessions: Changes you asked for

Remedy: A message indicates the survey was created and the URL to the survey is provided so that it can be posted anywhere.

2. More Cognitive Interviews

More Cognitive Interviews:

• Cognitive interviewing is a survey development technique that involves one-on-one interviews with a sample of the target population to assess the quality of survey questions.

• During the cognitive interview process, the interviewer uses techniques to elicit feedback on survey questions.

• We asked interviewees to read questions aloud and tell us what they thought the questions meant. Also, whether there was any terminology that was confusing.

More Cognitive Interviews:

• Original pilot study involved only 5 one-on-one interviews with undergraduate students

• Wanted to aim for 10 interviews prior to Fall 2010 when we planned to use the tool again in the classroom

Cognitive Interviews:Changes we made

Question:

• What are a few of the core databases for Education students?

  

Changes: • We changed “core databases” to “most important

databases” after we learned that some interviewees were confused by the original term.

• We changed “Education students” to “students in the College of Education” to be clearer in our meaning.

Cognitive Interviews:Changes we made

Question:

• When you start a search, why is it best not to limit results to full text only?

  

Changes: • We added “articles” after “full text” in the question after

we learned that some interviewees weren’t clear on the meaning of the phrase “full text.”

Cognitive Interviews:Changes we made

Question:

• In PsycInfo, how do I save a search result for the current search session?

  

Changes: • We made explicit that PsycInfo was a database, since

many of the interviewees did not know what the term meant.

Cognitive Interviews:Changes we made

Question:

• How can you tell you are reading a popular magazine?  

Changes: • Interviewees thought “popular” meant desirable or heavily

used material rather than non-scholarly so we changed the question to:

• How can you tell you are reading a magazine instead of a scholarly source of information?

3. Improved Statistical Analysis

Improved Statistical Analysis:Built in TTEST Reports

• We built in TTEST Reports through a link to an excel spreadsheet

Improved Statistical Analysis:Built in TTEST Reports

• Snapshot of the an excel spreadsheet report with TTEST tabulation

Improved Statistical Analysis:Advanced Correlation Templates

• Kendall’s Bivariate Correlation Analysis

• Asteriks by values in the matrix show statistically significant relationships between background factors.

• This data set shows that students who received prior library instruction had higher pre-scores than those who had not. Also those who received prior library instruction spent more time using library resources than those who had not.

4. Focus Group

Focus Group:Points for Discussion

• Are we testing teaching effectiveness or student learning?

• Do numbers offer the big picture of teaching effectiveness?

• Need more research to develop a cut score or acceptable percentage of correct responses.

• Need to gather support from other SJSU librarians and get campus endorsement

• The data is helpful to librarians and faculty – it gives us some ballpark idea of whether students are learning or not

Web Site Tool Kit:

URL:http://www.sjsu.edu/faculty/sstaley/sila/

• Provides complete overview of the entire assessment project from pilot study to present

• Complete data collected from usability testing sessions

• Complete data collected from focus group

Questions?