39
For Data Geeks…
For Data Geeks…
Connecting Assessment with Practice: Moving Information fromInteresting to Valuable
Darlena Jones, Ph.D.Director of Research and Development
Educational Benchmarking Inc
Assessment to Practice: A Strategy
• Assessment Instruments– Defining the goals of an assessment project– Designing assessment instruments– Identifying the various key stakeholders
• Reporting Strategies– Comprehending the range of reporting methods– Linking effective reporting methods and key institutional stakeholders– Discussing how effective reporting can promote changes in
institutional practice
• Examples in Action– MAP-Works Student Reporting– MAP-Works Faculty/Staff Reporting
Assessment Instruments
Assessment Instruments
Step 1: Understanding the Outcomes of the Assessment
Step 3: Use of Assessment Information
Step 2: Breadth or Depth of Assessment
Step 4: Appropriate
Survey Items
Keys to Successful
Assessment Instruments
Keys to a Successful Assessment Instrument
• Step 1: What are the outcomes of the assessment?– Program improvement?– Measuring the climate?– Measuring student learning?– Support accreditation / program review?
Keys to a Successful Assessment Instrument
• Step 2: What is the breadth or depth of your assessment?– To thoroughly educate your audience on a range of items
(diagnostic assessment)? • Strengths and weaknesses• Participant needs• Areas of improvement• Longitudinal trends• Implications and suggestions
– To quickly provide “just the facts” (targeted to a specific topic)?• Summaries• Focus on Outcomes
Keys to a Successful Assessment Instrument
• Step 3: What will your assessment inform?– Internal to Institution
• Decision making• Program evaluation and improvement• Budget allocations• Marketing and education• Interventions• Program development
– External to Institution• Accreditation reports• Grant reports• Benchmarking• Professional development• Publicity for alumni news, local media, etc.
Keys to a Successful Assessment Instrument
• Step 4: What questions are on your instrument?– Potential Problem: Survey is long and confusing
• Solution: Create survey that is focused on:– Improvement, not marketing– Performance, not activity (measure outputs, not activity)– But, most important, MISSION
– Potential Problem: Survey design is bad; factors have low reliability; results display poor validity
• Solution: Have “survey experts” design survey or involve people closely connected to issue help design survey
– Potential Problem: Don’t know where to begin improvement once analysis is finished
• Solution: Must have both analyses/systems that help guide action planning
Reporting Strategies
Reporting Strategies
Step 1: Understanding the Audience
Step 3: Using Valuable Information
Step 2: Appropriate Form for Information
Step 4: Choosing the Best Delivery
Method
Keys to Successful Reporting
Understanding the Audience• Step 1: Who will be reading your assessment report?
– Internal Stakeholders• Administrative decision-makers• Boards of trustees• Budget administrators• Faculty or staff• Internal governing bodies (Faculty/Staff Senate, Unions, etc.)• Students
– External Stakeholders• Accreditation bodies & reviewers• Alumni• Community members • Donors• Grant reviewers• Prospective students & parents• State and federal governments
Understanding the Audience
• Step 1 (cnt): What is your audience’s experience with information?– Quantitative vs. Qualitative
• How comfortable are they with statistics?• Do they prefer narratives or numbers?
– Interest and experience - How much explanation regarding the…• Topic?• Assessment methods?• Results?• Implications?
– Time available - How much time can or will they spend reading the results?
– Level - Will they use university level data? College level? Department level? Individual level?
Appropriate Form for Information
• Step 2: What is the best form for the information?– Types of reports
• Executive summaries• Comprehensive reports• Assessment summaries• Notes, brochures, flyers, and memos• Institutional snapshots• Interactive data
– Easily read?• What does the report look like (size of font, appearance, visuals,
etc.)?• Do they want to read this? Does it draw them in? Does it intimidate
them or overwhelm them?
Using Valuable Information
• Step 3: What is the content of the report?– Importance: Does this report…
• Include important issues?• Highlight and emphasize the important results?• Differentiate between important and non-important results?
– Usefulness: Does this report…• Discuss the implications of the results?• Clearly link the results to practice?• Help practitioners determine what should be done?• Differentiate between useful results and interesting results?
Choosing the Best Delivery Method
• Step 4: How will you deliver the assessment results?– Media formats
• Paper• Electronic (websites, downloadable files, CDs, emails, etc.)• Oral presentations• Combinations
– Easily accessible?• How hard is the information for Stakeholders to access or find?• Can they find what they need quickly?
Examples in Action: MAP-Works Student Report
EBI’s “best practices” model of reporting
Who is Responsible?• Who is responsible for student
success on your campus?– Enrollment Management/
Retention?– Student Affairs?– Academic Affairs?
• What information do you know about this first-year student?
Student ID: YD252952HS GPA: 3.93SAT Verbal: 29Location: In stateGender: FemaleRace: African AmericanAge: 18Major: Undecided
Do you really know them?
Enrollment Management/R
etentionStudent Affairs
Academic Affairs
Paradigm Shift• What would happen if…
– ALL faculty/staff were responsible for student success?
– YOU knew student was struggling?
• Could you do something about it before it was too late?
Enrollment Management /
Retention
Residence Hall Staff
Academic Advisor
First-Year Seminar Instructor
Academic Department
Heads
Financial Aid
Minority Student Affairs
I’m really homesick
I don’t think I can afford college My roommate
and I argue all the time
Student AffairsAcademic
Affairs
I’m thinking about
transferring
I’m struggling in my math
class
MAP-Works Mission – 4 WayAcademic Success: Improve students' ability to succeed academically by realigning behavior with grade expectations and focusing on elements of academic success
Retention: Minimize percentage of capable students who drop out due to issues that could have been addressed by self-awareness or timely intervention by staff/faculty
Student Development:
Facilitate the establishment of
relationships, address
homesickness, identify residence
hall living issuesStudent Involvement: Connect students with campus resources to facilitate involvement with student organizations and campus programming
MAP-Works History
1988, Ball State developed concept
2005, Ball State partnered with EBI to create MAP-Works
1989 to 2004, Ball State used MAP in-house
Fall 2008 -approximatel
y 40 campuses
will use MAP-Works!
• In 1988, Ball State had a number of concerns…
– First-year students arrived with unrealistic expectations (academics, grades, housing, etc.)
– Retention rates were not as high as they wanted them to be
– Faculty and staff were concerned that identifying student issues at mid-term was too late
– Faculty and staff wanted better data about incoming students
MAP-Works Process
•Expectations
•Behaviors
•Social Norming
•Expectations
•CampusResources
•Student Summary
•Scan Students
•Student Profile•Institution Profile•Campus
Resources
Understanding the Student’s Experience
Survey and Profile Items
• Academic Integration– Academic Self-Efficacy– Core Academic Behaviors– Advanced Academic Behaviors– Commitment to Higher Education
• Self-Assessment– Communication Skills– Analytical Skills– Personal Management– Time Management– Health and Wellness– Potential Issues (stress, financial,
etc.)
• Profile Information– Student information like gender and
race/ethnicity– Entrance exam scores– # credit hours enrolled
• Social Integration– Campus Relationships– Living Environment (on/off campus)– Roommate Relationships– Homesickness
• Who benefits from MAP-Works?– First-Year Students– Departments like…
• Housing & Residence Life• Academic Advising• Enrollment Management / Retention• First-Year Seminar Instructors• Minority Student Affairs• Athletic Department• Student Activities
– Academic Assessment & Institutional Research– Upper Administration
Who Benefits from MAP-Works?
MAP-Works Student Reports
• Student Reporting– Individualized on-line report provided directly to students within days of
assessment
• 3 Main Reporting Purposes– Purpose 1: Realign expectations– Purpose 2: Information to help them plan for their success– Purpose 3: Connect with appropriate campus resources
Purpose 1: Realign Expectations (Example 1)
Report provides benchmark information to help students realign
expectations
REPORTING TIP: Only provide the most valuable information in an summary report. Keep the “interesting” information for the larger report.
Purpose 1: Realign Expectations (Example 2)
Report provides benchmark information to help students realign
expectations
REPORTING TIP: Know your audience! Consider using very simple charts to relay complex information (most students don’t have experience reading complex charts/tables).
Purpose 2: Plan for Success
Report provides feedback to help student understand need to plan for their future success
REPORTING TIP: Consider providing written explanations to help the reader draw conclusions.
Purpose 3: Connect with Resources
School lists campus resources/offices
that link with reporting area
REPORTING TIP: Consider providing additional information the reader may want to access.
Evaluating MAP-Works Student Reporting
Step 1: Understanding the Audience – Students have less experience with information; reporting is easy to read
Step 3: Using Valuable Information - Targeted to helping them be successful in school
Step 2: Appropriate Form for Information –Extremely visual and colorful. Minimal information per slide.
Step 4: Choosing the Best Delivery
Method – Web based is a delivery
method most expected by this
generation
Keys to Successful Reporting
Examples in Action: MAP-Works Faculty/Staff Report
MAP-Works Faculty/Staff Reporting
• Interactive system for faculty and staff– Residence hall staff – Academic advisors– First-Year Seminar Instructors– Other staff positions?
• 3 Main Reporting Purposes– Purpose 1: Identify students who may benefit from personalized
attention – Purpose 2: Provide information for one-on-one meetings with students– Purpose 3: Provide input regarding programming and training needs
Purpose 1: Identifying Students (Example 1)
These students are not adjusting academically or socially and are not committed to their institution.
REPORTING TIP: Consider creating high level “dashboards” that provides a lot of information in an easy-to-read format.
Purpose 1: Identifying Students (Example 2)
These students very interested in leadership positions – Recruit them!
These students are moderately interested in a leadership position – Convince them?
These students definitely not interested in a leadership position – Why? Important on résumé
REPORTING TIP: Consider using simple color codes instead of statistical information to relay information.
Purpose 2: Individual Meetings (Example 1)
Kimberly is having issues adjusting to college (both academically and socially) and early warning indicators are not good.
REPORTING TIP: Consider creating “dashboard” reports that give a 30,000 foot look at the data.
Purpose 2: Individual Meetings (Example 2)
REPORTING TIP: Consider providing easy access to in-depth information if the reader needs more clarification.
Purpose 3: Programming
These students have self rated themselves low in Public Speaking skills.
How could this be addressed through programming?
REPORTING TIP: Consider that the information you provide can be used in multiple ways and how that information can be presented to best tell the story.
Evaluating MAP-Works Faculty/Staff Reporting
Step 1: Understanding the Audience – Reporting is color-coded for readability; No special training required
Step 3: Using Valuable Information – Highlights students with more critical issues; dashboards are focused on critical transition
issues
Step 2: Appropriate Form for Information –Extremely visual and colorful. Easy navigation.
Step 4: Choosing the Best Delivery
Method – Web based is a delivery
method that provides most
flexibility in searching through
information
Keys to Successful Reporting
Additional Questions and Discussion…
Darlena Jones, Ph.D.Director of Research and Development
Educational Benchmarking [email protected]
For more information about MAP-Works, contactTodd Pica, [email protected], visit www.MAP-Works.com
