For Data Geeks…

transcript

For Data Geeks…

Connecting Assessment with Practice: Moving Information fromInteresting to Valuable

Darlena Jones, Ph.D.Director of Research and Development

Educational Benchmarking Inc

Assessment to Practice: A Strategy

• Assessment Instruments– Defining the goals of an assessment project– Designing assessment instruments– Identifying the various key stakeholders

• Reporting Strategies– Comprehending the range of reporting methods– Linking effective reporting methods and key institutional stakeholders– Discussing how effective reporting can promote changes in

institutional practice

• Examples in Action– MAP-Works Student Reporting– MAP-Works Faculty/Staff Reporting

Assessment Instruments

Step 1: Understanding the Outcomes of the Assessment

Step 3: Use of Assessment Information

Step 2: Breadth or Depth of Assessment

Step 4: Appropriate

Survey Items

Keys to Successful

Assessment Instruments

Keys to a Successful Assessment Instrument

• Step 1: What are the outcomes of the assessment?– Program improvement?– Measuring the climate?– Measuring student learning?– Support accreditation / program review?

• Step 2: What is the breadth or depth of your assessment?– To thoroughly educate your audience on a range of items

(diagnostic assessment)? • Strengths and weaknesses• Participant needs• Areas of improvement• Longitudinal trends• Implications and suggestions

– To quickly provide “just the facts” (targeted to a specific topic)?• Summaries• Focus on Outcomes

• Step 3: What will your assessment inform?– Internal to Institution

• Decision making• Program evaluation and improvement• Budget allocations• Marketing and education• Interventions• Program development

– External to Institution• Accreditation reports• Grant reports• Benchmarking• Professional development• Publicity for alumni news, local media, etc.

• Step 4: What questions are on your instrument?– Potential Problem: Survey is long and confusing

• Solution: Create survey that is focused on:– Improvement, not marketing– Performance, not activity (measure outputs, not activity)– But, most important, MISSION

– Potential Problem: Survey design is bad; factors have low reliability; results display poor validity

• Solution: Have “survey experts” design survey or involve people closely connected to issue help design survey

– Potential Problem: Don’t know where to begin improvement once analysis is finished

• Solution: Must have both analyses/systems that help guide action planning

Reporting Strategies

Step 1: Understanding the Audience

Step 3: Using Valuable Information

Step 2: Appropriate Form for Information

Step 4: Choosing the Best Delivery

Method

Keys to Successful Reporting

Understanding the Audience• Step 1: Who will be reading your assessment report?

– Internal Stakeholders• Administrative decision-makers• Boards of trustees• Budget administrators• Faculty or staff• Internal governing bodies (Faculty/Staff Senate, Unions, etc.)• Students

– External Stakeholders• Accreditation bodies & reviewers• Alumni• Community members • Donors• Grant reviewers• Prospective students & parents• State and federal governments

Understanding the Audience

• Step 1 (cnt): What is your audience’s experience with information?– Quantitative vs. Qualitative

• How comfortable are they with statistics?• Do they prefer narratives or numbers?

– Interest and experience - How much explanation regarding the…• Topic?• Assessment methods?• Results?• Implications?

– Time available - How much time can or will they spend reading the results?

– Level - Will they use university level data? College level? Department level? Individual level?

Appropriate Form for Information

• Step 2: What is the best form for the information?– Types of reports

• Executive summaries• Comprehensive reports• Assessment summaries• Notes, brochures, flyers, and memos• Institutional snapshots• Interactive data

– Easily read?• What does the report look like (size of font, appearance, visuals,

etc.)?• Do they want to read this? Does it draw them in? Does it intimidate

them or overwhelm them?

Using Valuable Information

• Step 3: What is the content of the report?– Importance: Does this report…

• Include important issues?• Highlight and emphasize the important results?• Differentiate between important and non-important results?

– Usefulness: Does this report…• Discuss the implications of the results?• Clearly link the results to practice?• Help practitioners determine what should be done?• Differentiate between useful results and interesting results?

Choosing the Best Delivery Method

• Step 4: How will you deliver the assessment results?– Media formats

• Paper• Electronic (websites, downloadable files, CDs, emails, etc.)• Oral presentations• Combinations

– Easily accessible?• How hard is the information for Stakeholders to access or find?• Can they find what they need quickly?

Examples in Action: MAP-Works Student Report

EBI’s “best practices” model of reporting

Who is Responsible?• Who is responsible for student

success on your campus?– Enrollment Management/

Retention?– Student Affairs?– Academic Affairs?

• What information do you know about this first-year student?

Student ID: YD252952HS GPA: 3.93SAT Verbal: 29Location: In stateGender: FemaleRace: African AmericanAge: 18Major: Undecided

Do you really know them?

Enrollment Management/R

etentionStudent Affairs

Academic Affairs

Paradigm Shift• What would happen if…

– ALL faculty/staff were responsible for student success?

– YOU knew student was struggling?

• Could you do something about it before it was too late?

Enrollment Management /

Retention

Residence Hall Staff

Academic Advisor

First-Year Seminar Instructor

Academic Department

Financial Aid

Minority Student Affairs

I’m really homesick

I don’t think I can afford college My roommate

and I argue all the time

Student AffairsAcademic

Affairs

I’m thinking about

transferring

I’m struggling in my math

MAP-Works Mission – 4 WayAcademic Success: Improve students' ability to succeed academically by realigning behavior with grade expectations and focusing on elements of academic success

Retention: Minimize percentage of capable students who drop out due to issues that could have been addressed by self-awareness or timely intervention by staff/faculty

Student Development:

Facilitate the establishment of

relationships, address

homesickness, identify residence

hall living issuesStudent Involvement: Connect students with campus resources to facilitate involvement with student organizations and campus programming

MAP-Works History

1988, Ball State developed concept

2005, Ball State partnered with EBI to create MAP-Works

1989 to 2004, Ball State used MAP in-house

Fall 2008 -approximatel

y 40 campuses

will use MAP-Works!

• In 1988, Ball State had a number of concerns…

– First-year students arrived with unrealistic expectations (academics, grades, housing, etc.)

– Retention rates were not as high as they wanted them to be

– Faculty and staff were concerned that identifying student issues at mid-term was too late

– Faculty and staff wanted better data about incoming students

MAP-Works Process

•Expectations

•Behaviors

•Social Norming

•Expectations

•CampusResources

•Student Summary

•Scan Students

•Student Profile•Institution Profile•Campus

Resources

Understanding the Student’s Experience

Survey and Profile Items

• Academic Integration– Academic Self-Efficacy– Core Academic Behaviors– Advanced Academic Behaviors– Commitment to Higher Education

• Self-Assessment– Communication Skills– Analytical Skills– Personal Management– Time Management– Health and Wellness– Potential Issues (stress, financial,

• Profile Information– Student information like gender and

race/ethnicity– Entrance exam scores– # credit hours enrolled

• Social Integration– Campus Relationships– Living Environment (on/off campus)– Roommate Relationships– Homesickness

• Who benefits from MAP-Works?– First-Year Students– Departments like…

• Housing & Residence Life• Academic Advising• Enrollment Management / Retention• First-Year Seminar Instructors• Minority Student Affairs• Athletic Department• Student Activities

– Academic Assessment & Institutional Research– Upper Administration

Who Benefits from MAP-Works?

MAP-Works Student Reports

• Student Reporting– Individualized on-line report provided directly to students within days of

assessment

• 3 Main Reporting Purposes– Purpose 1: Realign expectations– Purpose 2: Information to help them plan for their success– Purpose 3: Connect with appropriate campus resources

Purpose 1: Realign Expectations (Example 1)

Report provides benchmark information to help students realign

expectations

REPORTING TIP: Only provide the most valuable information in an summary report. Keep the “interesting” information for the larger report.

Purpose 1: Realign Expectations (Example 2)

Report provides benchmark information to help students realign

expectations

REPORTING TIP: Know your audience! Consider using very simple charts to relay complex information (most students don’t have experience reading complex charts/tables).

Purpose 2: Plan for Success

Report provides feedback to help student understand need to plan for their future success

REPORTING TIP: Consider providing written explanations to help the reader draw conclusions.

Purpose 3: Connect with Resources

School lists campus resources/offices

that link with reporting area

REPORTING TIP: Consider providing additional information the reader may want to access.

Evaluating MAP-Works Student Reporting

Step 1: Understanding the Audience – Students have less experience with information; reporting is easy to read

Step 3: Using Valuable Information - Targeted to helping them be successful in school

Step 2: Appropriate Form for Information –Extremely visual and colorful. Minimal information per slide.

Method – Web based is a delivery

method most expected by this

generation

Examples in Action: MAP-Works Faculty/Staff Report

MAP-Works Faculty/Staff Reporting

• Interactive system for faculty and staff– Residence hall staff – Academic advisors– First-Year Seminar Instructors– Other staff positions?

• 3 Main Reporting Purposes– Purpose 1: Identify students who may benefit from personalized

attention – Purpose 2: Provide information for one-on-one meetings with students– Purpose 3: Provide input regarding programming and training needs

Purpose 1: Identifying Students (Example 1)

These students are not adjusting academically or socially and are not committed to their institution.

REPORTING TIP: Consider creating high level “dashboards” that provides a lot of information in an easy-to-read format.

Purpose 1: Identifying Students (Example 2)

These students very interested in leadership positions – Recruit them!

These students are moderately interested in a leadership position – Convince them?

These students definitely not interested in a leadership position – Why? Important on résumé

REPORTING TIP: Consider using simple color codes instead of statistical information to relay information.

Purpose 2: Individual Meetings (Example 1)

Kimberly is having issues adjusting to college (both academically and socially) and early warning indicators are not good.

REPORTING TIP: Consider creating “dashboard” reports that give a 30,000 foot look at the data.

Purpose 2: Individual Meetings (Example 2)

REPORTING TIP: Consider providing easy access to in-depth information if the reader needs more clarification.

Purpose 3: Programming

These students have self rated themselves low in Public Speaking skills.

How could this be addressed through programming?

REPORTING TIP: Consider that the information you provide can be used in multiple ways and how that information can be presented to best tell the story.

Evaluating MAP-Works Faculty/Staff Reporting

Step 1: Understanding the Audience – Reporting is color-coded for readability; No special training required

Step 3: Using Valuable Information – Highlights students with more critical issues; dashboards are focused on critical transition

issues

Step 2: Appropriate Form for Information –Extremely visual and colorful. Easy navigation.

Method – Web based is a delivery

method that provides most

flexibility in searching through

information

Additional Questions and Discussion…

Darlena Jones, Ph.D.Director of Research and Development

Educational Benchmarking IncDarlena@webebi.com

For more information about MAP-Works, contactTodd Pica, Todd@webebi.comOr, visit www.MAP-Works.com

For Data Geeks…

Documents