Developing a QEP Assessment Plan: Criteria and Process of Instrument Review and Development
M. David Miller, Director, QEPTimothy S. Brophy, Director, Institutional AssessmentUniversity of Florida
Reaffirmation 2014
Off-site review is completed
On-site review is February 2014
QEP is ready to be submitted in January 2014
University of Florida Accreditation Process
The institution has developed an acceptable Quality Enhancement Plan that includes an institutional process for identifying key issues emerging from institutional assessment and focuses on learning outcomes and/or the environment supporting student learning and accomplishing the mission of the institution. (Quality Enhancement Plan)
Core Requirement 2.12
The Institution has developed a Quality Enhancement Plan that (1) demonstrates institutional capability for the initiation, implementation and completion of the QEP; (2) includes broad-based involvement of institutional constituencies in the development and proposed implementation of the QEP; and (3) identifies goals and a plan to assess their achievement.
Standard 3.3.2
Purpose
Uses and interpretations
Studen
t
Learnin
g
(Outco
mes)
Begin with the end in mind
QEP Assessment
Step 1. Establish topic and student learning outcomes (SLOs)
Step 2. Identify direct and indirect assessments to measure SLOs (and outputs to measure initiatives)
Step 3. If assessments are not available and sufficient, develop your own assessments
Step 4. Plan Assessment Schedule and follow it
Procedure for Planning Assessment
Direct – Measures of Student Learning
Indirect – Measures of Attitudes, Behaviors, Beliefs, …
Outputs – Counts of the numbers of students participating in events, number of events, number of classes, etc.
Types of Assessments
Direct and indirect assessments must measure progress on the SLOs
Outputs must measure participation in the initiatives
Each measure must have a planned use or interpretation (Validity)
Each measure must be reliably measured (with minimum error)
Our Guiding Principles
It is easier to adopt an existing instrument than to develop a new instrument
Easier is not better if you violate your principles!
That is, existing instruments must: measure the SLOs; be valid for its planned uses and
interpretations; and be reliable for your purposes
Our Guiding Principles, Part II
UF Theme : Internationalization
Internationalization is the conscious integration of global awareness and intercultural competence into student learning.
Step 1: Define Theme and SLOs
SLO1 (Content):Students identify, describe, and explain global and intercultural conditions and interdependencies.
SLO2 (Critical Thinking): Students analyze and interpret global and intercultural issues.
SLO3 (Communication): Students communicate effectively with members of other cultures.
Step 1: Define Theme and SLOs
Internationalization Measures Identified – Commercially Available
◦ Global Perspectives Inventory (GPI)◦ Global Competence Aptitude Assessment
(GCAA)◦ Intercultural Development Inventory (IDI)◦ Global Competencies Inventory◦ Cross Cultural Adaptability Inventory◦ Global Awareness Profile◦ Intercultural Effectiveness Scale
Step 2: Identify Existing Instruments
International Task Force◦ Representation from all 16 Colleges◦ Student Representation◦ Representation from Faculty, Staff, Students,
Administration
Assessment Committee (5-6 with expertise on content and assessment)
Step 2a: Identify Experts for Review
Measures the SLOs
Validity Evidence
Reliability Evidence
Feasibility to Use in Our Context
Step 2b: Defining Review Criteria
Most commercial products had good reliability and validity evidence for their stated purposes
None matched our SLOs
Little evidence of feasibility for large scale use
Final Decision: Need to develop our own assessments
Step 2c: Factors in our decision to create our own assessments
Consider Options Already Being Used as Part of Reporting System
Student Experience in the Research University (SERU) - biennial survey used by multiple universities that includes items on internationalization
Existing Surveys on Campus
Items measure behaviors (e.g., courses taken, participation in Study Abroad, and other types of international experiences) and attitudes toward other cultures
We elected to add 10 items designed to
measure attitudes related to SLOs 2 and 3
SERU
Direct Assessments focusing on SLO achievement
Must allow flexibility to measure learning
in any discipline
Needs to be aligned with Curriculum
Indirect Assessments of SLOs 2 and 3 only
Not appropriate for SLO 1 since Content is discipline-based
What Assessments do we need to Develop?
Writing the Assessment Items
Ongoing review for fidelity with the SLOs (Validity)
Piloting to establish reliability and psychometric properties of items
Developing the Assessments
• International Critical Thinking (IntCRIT) Attitudes and Beliefs
• International Communication (IntCOMM) Attitudes and Beliefs
• International Content not feasible with Disciplinary Differences
Two Assessments Developed
Step 3a: Indirect Assessments
1.Development of item specifications based on the two SLOs and a literature review.
2.Writing items based on the item specifications.
Approximately 70 items were written for each SLO
Indirect Assessment Development Steps
3.Review of the items by the ITF, the Assessment Committee and other
experts in assessment. (Validity and Match to SLOs)
4. Revision of items based on feedback from expert review.
Revisions were minor changes in wording.
Indirect Assessment Development Steps
5. Pilot testing with undergraduate students at UF and eliminating items with poor discriminations.
◦ Initial piloting was completed with four forms to minimize the testing burden for students.
◦ Forms overlapped with ten items that expert agreed helped to define the construct.
◦ Each form was pilot tested with 70-100 undergraduates.
6. Item analysis of pilot data. ◦ The scale reliabilities exceeded .95 for all four forms.◦ Items were retained that had an item discrimination
of .25 or higher.
Indirect Assessment Development Steps
7. Pilot testing (N=70-80) the retained items on a single form for each SLO.
8. Item analysis of pilot data. ◦ Recommended retaining items with the highest item discriminations
that would result in a scale with a reliability of at least .90. For IntCRIT, the recommendation was to retain 12 items. For IntCOMM, the recommendation was to retain 14 items.
9. Review of the items by the ITF, the Assessment Committee and other experts in assessment. (Validity and Match to SLOs)
Indirect Assessment Development Steps
1. I consider different perspectives before making conclusions about the world.
2. I am able to manage when faced with multiple cultural perspectives.3. I am open to different cultural ways of thinking in any international
context.4. I can make effective decisions when placed in different cultural
situations5. Knowing about other cultural norms and beliefs is important to me.6. I am able to think critically to interpret global and intercultural
issues.7. I actively learn about different cultural norms.8. Understanding different points of view is a priority to me.9. I can recognize how different cultures solve problems.10. I can contrast important aspects of different cultures with my own.11. Knowing about other cultural beliefs is important.12. I am able to recognize how members of other cultures make
decisions
IntCRIT
1. I demonstrate flexibility when interacting with members of another culture.
2. I prefer to socialize with people of my culture.3. I am confident that I can adapt to different cultural environments4. I am able to communicate effectively with members of other cultures5. I like working in groups with students from other countries.6. I feel comfortable in conversations that may involve cultural
differences.7. When working on a group project, I enjoy collaborating with students
from other countries.8. I often ask questions about culture to members of other cultures.9. I enjoy learning about other cultures10. I appreciate members of others cultures teaching me about their
culture.11. I am able to interact effectively with members of other cultures.12. I appreciate differences between cultures13. I feel comfortable discussing international issues.14. I can clearly articulate my point of view to members of other cultures
IntCOMM
Use with curriculum
Must be flexible to allow faculty to define internationalization in discipline
Must provide a standard measure
Step 3b: Direct Assessments
Define Rubric
Instructors develop or identify assessments in courses
Instructors will provide evidence of use and scores to University with examples of assessments
Adopt General Education model on campus
The Association of American Colleges and Universities developed 15 Rubrics that can be used across programs and courses
VALUE (Valid Assessment of Learning in Undergraduate Education) rubrics were developed by faculty and assessment expert teams across the country -- used by more than 2000 institutions (http://www.aacu.org/value/index.cfm)
Developing Rubrics
UF QEP SLO VALUE Rubric Adaptation
SLO 1: Content
Intercultural Knowledge and Competence
Limit criteria to knowledge, dropping skills and attitudes. Modify descriptions for consistency across levels and ease of use.
SLO 2: Critical Thinking
Critical Thinking
Add language to reflect emphasis on international context for critical thinking. Modify descriptions for consistency across levels and ease of use.
SLO 3: Communication
Written Communication
Oral Communication
Combine rubrics to measure communication in multiple modes, andadd language to reflect emphasis on international context. Modify descriptions for consistency across levels and ease of use.
Adapting VALUE Rubrics for SLOs
QEP Content Rubric
SLOComponents
Outstanding3
Satisfactory2
Unsatisfactory1
Not Applicable0
Concepts/Principles
Consistently and effectively demonstrates sophisticated understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, and beliefs and practices.
Usually demonstrates understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices.
Rarely or never understands the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices.
Not Applicable to Assignment or Course
Terminology
Consistently recognizes and effectively utilizes important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.
Usually identifies and implements important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.
Rarely or never understands important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.
Not Applicable to Assignment or Course
Methodologies
Consistently comprehends and effectively utilizes diverse and appropriate methodologies for understanding complex intercultural and global issues.
Usually comprehends and utilizes diverse and appropriate methodologies for understanding intercultural and global issues.
Rarely or never comprehends and utilize diverse and appropriate methodologies for understanding intercultural and global issues.
Not Applicable to Assignment or Course
Review by Internationalization Task Force, Assessment Committee, and Experts for Match to SLOs and Validity
Piloting (in progress) in curriculum and Study Abroad
Additional Steps
Number of participants at specific campus events with an international focus (QEP events).
Number of International Scholar courses. Number of students enrolled in International
Scholar courses. Number of Study Abroad courses offered. Number of students studying abroad.
Outputs
University wide impact
Indirect Assessments
SERU Biennial, odd years
IntCrit and IntComm – Annual – Fall -
Sampling
Outputs
Monitor participation and implementation -
Annual
Impact on those taking courses or
Study Abroad
Direct Assessments - Rubrics
Indirect Assessments –
IntCrit and IntComm
Output – Number of students and
options
Assessment Plan
Annual analysis, review, and
interpretation of data to inform:
The implementation of the program
The degree of SLO achievement
When adjustments to program are
needed
The benefit of the program for
underrepresented populations
Closing the Loop
