PAR Framework Review and Summary

Ellen Wagner, chief strategy officer, PAR Framework

Beth Davis, managing directorPAR Framework

Data Are Changing Everything

Data and Evidence basedDecision making in Higher Ed

Analytics have ramped up everyone’s expectations ofpersonalization, accountability and transparency.Academic enterprises cannot live outside theinstitutional focus on tangible, measurable resultsdriving IT, finance, recruitment and other missioncritical concerns.

Costs and Completion Rates

Source: New York Times; NCES

0

10

20

30

40

50

60

70

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2 yr colleges4 yr colleges

Graduation rates at 150% of time

Cohort year

Performance Based Funding

http://www.ncsl.org/issues research/educ/performance funding.aspx

Institutional Accountability

http://www.whitehouse.gov/issues/education/higher education/college score card

While Big Data raise expectations,student data drive big decisions in .edu

PAR Framework video

• https://www.dropbox.com/s/jksxbc6uac4n1rg/PAR_pre delivery.mp4

PAR Outputs

IdentifyShow how institutionscompare to their peers instudent outcomes, byscaling amultiinstitutional databasefor benchmarking andresearch purposes.

Target

Identify which studentsneed assistance, by usingin depth, institutionalspecific predictive models.

Models are unique to theneeds and priorities of ourmember institutions basedon their specific data.

Determine best ways toaddress weaknessesidentified in benchmarksand models by scalingand leveraging amember, data andliterature validatedframework for examininginterventions within andacross institutions(SSMx)

Treat

PRELIMINARY UND FINDINGS FROM PAR, SPRING, 2014

Beth DavisManaging DirectorPAR Framework

Rapid Results

• Data Delivery– First data meeting 1/24– Preliminary data provided by 2/18– Final data expected 3/24

• Early discovery– UND course catalog– Basic student information file– Did not include financial aid, credentials

With limited data set and in less than 1 week

Identified• The key gatekeeper courses• With over 70% accuracy the likelihoodstudents will succeed in college level courses *

• 10 most predictive factors known aboutstudent at entry.

* Accuracy will increase when data is complete, reaches 80 90+% for many scenarios

Predictions

• Everyone who started fall 2013.• A prediction of them taking one of thecourses.

• Representative of early success in a collegelevel course.

Course name and course success Key factors include:HS GPAHS credits by examTransfer CreditsRaceGender

HS GPA >3.8

HS GPA < 2.6

Top Risk Factors

• HS GPA less than 2.65• GED• Race (B, AI, H)• Age at start ><19• Gender• Transfer status

Demographics that impact student success

Risk Factors • Students who had a GPA between 0 and 2.66failed the gateway courses 57% of the time.

Risk ratios • The factors where variation impacts studentsuccess and by how much

• Students with a HS GPA not specified are 4X lesslikely to succeed in college level courses

WatchlistPARanonymized ID 1st, 2nd and 3rd most important

factors contributing to risk

Risk they will notsucceed in collegelevel course

DISCUSSION

1229Thomas M. DiLorenzo, Ph.D. • Exceptional UND: A Critical Step in Realizing the Vision April 22, 2014 • University of North Dakota

Performance metricsInformed decision-making forStudent Success Systems

1230Thomas M. DiLorenzo, Ph.D. • Exceptional UND: A Critical Step in Realizing the Vision April 22, 2014 • University of North Dakota

OUR APPROACH

• Progress can be made quickly• Graduation rates are everyone s responsibility• We need data-based approaches and• Proactive/prevention systems• Focus part of your time on students through systems and

processes that provide efficient, accurate, and integrated support

• And part of your time on at-risk students, identified through custom systems

1231Thomas M. DiLorenzo, Ph.D. • Exceptional UND: A Critical Step in Realizing the Vision April 22, 2014 • University of North Dakota

EFFICIENT, ACCURATE AND INTEGRATED SYSTEMS

• Online Catalog• Standard Schedule• Communications Strategy• Degree Audit/Planner• One Stop Student Services Center

1232Thomas M. DiLorenzo, Ph.D. • Exceptional UND: A Critical Step in Realizing the Vision April 22, 2014 • University of North Dakota

SYSTEMS/STRUCTURAL CHANGES (EXAMPLES)

• Major selection• 120-90-75-60 Initiative• Four year graduation plans

1233Thomas M. DiLorenzo, Ph.D. • Exceptional UND: A Critical Step in Realizing the Vision April 22, 2014 • University of North Dakota

PROACTIVE INTERVENTIKONWITH AT-RISK STUDENTS

• Faculty Early Alert System• Advisor CRM (Client Relationship

Management)• Advisor Utilization• Departed Student Outreach• Advanced Data Analysis

1234Thomas M. DiLorenzo, Ph.D. • Exceptional UND: A Critical Step in Realizing the Vision April 22, 2014 • University of North Dakota

ASSESS AT-RISK /DEVELOP INTERVENTIONS

• Partnership between Academic Affairs, Student Affairs and Financial Affairs

• Faculty involvement in all phases• Student involvement in all phases• Dean and Chairs involvement in all phases

1235Thomas M. DiLorenzo, Ph.D. • Exceptional UND: A Critical Step in Realizing the Vision April 22, 2014 • University of North Dakota

1236Thomas M. DiLorenzo, Ph.D. • Exceptional UND: A Critical Step in Realizing the Vision April 22, 2014 • University of North Dakota

EXAMPLES

• Undergraduate research• Internships• Service learning• Study abroad• Undergraduate scholarships/fellowships• Honors experiences

The Predictive Analytics Reporting (PAR) Framework

• PAR is a national, non profit multi institutionalcollaborative focused on institutional effectiveness andstudent success.

• PAR is a “big data” analysis effort using predictiveanalytics to identify drivers related to loss and momentumand to inform student loss prevention

• PAR member institutions voluntarily contribute deidentified student records to create a single federateddatabase.

• Descriptive, inferential and predictive analyses have beenused to create benchmarks, institutional predictive modelsand to map student success interventions to predictorbehaviors

Common Definitions Lead to Shared Understanding

Analysis/Modeling Process

• Analysis and model building is aniterative process

• Around 70 80% efforts are spenton data exploration andunderstanding.

Structured, Readily Available Data• Common data

definitions = reusablepredictive models andmeaningfulcomparisons.

• Openly published via acc license @https://public.datacookbook.com/public/institutions/par

PAR Data Inputs Student

Demographics& Descriptive

GenderRace

Prior CreditsPerm Res Zip CodeHS InformationTransfer GPAStudent Type

Student CourseStudent CourseInformationCourse Location

SubjectCourse Number

SectionStart/End DatesInitial/Final GradeDelivery ModeInstructor StatusCourse Credit

StudentAcademicProgress

Curent Major/CIPEarned Credential/CIP

StudentFinancial

InformationFAFSA on File – Date

Pell Received/Awarded –Date

Course CatalogSubject

Course NumberSubject LongCourse Title

Course DescriptionCredit Range

** Future

Lookup TablesCredential Types OfferedCourse Enrollment Periods

Student TypesInstructor StatusDelivery ModesGrade Codes

Institution Characteristics

Possible Additional **Placement TestsNSC InformationSES Information

Satisfaction SurveysCollege Readiness SurveysIntervention Measures

PAR OutputsDescriptiveBenchmarks

Show how institutionscompare to their peers instudent outcomes, byscaling amultiinstitutional databasefor benchmarking andresearch purposes.

PredictiveModels

Identify which studentsneed assistance, by usingin depth, institutionalspecific predictive models.Models are unique to theneeds and priorities of ourmember institutions basedon their specific data.

Institutions addressareas of weaknessidentified inbenchmarks and modelsby scaling and leveraginga member, data andliterature validatedframework for examininginterventions within andacross institutions(SSMx)

InterventionMatrix

Feedback loops for enabling institutional performance improvements

PerformanceBenchmarksPerformanceBenchmarks

InterventionBenchmarksInterventionBenchmarks

PredictiveModelsPredictiveModelsActionAction

MeasurableResults

MeasurableResults

CommonData

Definitionsand DataWarehouse

Scalable cross institutional improvements enabled byCollaboration via PAR

Benchmarking in risk factor areas

PAR Student Success Matrix (SSMx)

Literature based tool forbenchmarking student services

and interventions 600+ total interventions submittedAbility to compare among all 16 PAR institutions Basis for institutional intervention field tests Publically available, over 1,000 downloads since June 2013

https://par.datacookbook.com/public/institutions/par

Student Success Framework Identified 38 distinct functionalcategories of interventions

• Mapped >600 interventionsto functional categories foreasy benchmarking andcomparisons

Literature and partner validatedpredictors reveal 80 risk factors

• Tying interventions to thepredictive factors enablesinsight into how to bestapply institutional resources

• level of intervention (student, course, section, program,institution, . . .)

• focus (audience)

• delivery channels

• impact (# students affected)

• results measurement (outcomes, if available)

• Return on Investment (ROI)

©PAR Framework 2013

SSMX ONLINE APPLICATION

Summary, Conclusions

Thank You