Post on 22-Jan-2018
transcript
Confronting Data Manipulation on Your
Campus
Dr. Will Miller
Director of Institutional Research and Effectiveness
Flagler College
Context
• We rely on others for data• IPEDS
• Outcomes reports
• Closing the Loop
• Course evaluations
• Technology has eliminated the paper trail
Data Manipulation
• Comes in a variety of flavors
• Different than data error!• Survey responses
• Course evaluations
• Course outcomes
• Closing the Loop reports
• Or more troubling forms
Flagler College
• Founded in 1968
• Baccalaureate only
• Approximately 2700 students
• U.S. News Top Regional
• The unicorn of admissions• Increase applicant pool
• Increase cohort size
• Decrease merit scholarships
• Increase class profile
But…
• Unicorns aren’t real
The Swift Fall of the Unicorn
• Background of the offender
• New Director of IRE and New Director of College Writing in Summer 2013
• During a random audit of student placements from fall semester, we notice a vast majority of students were under-placed in writing courses according to information in Student Management System
• Initially blame advisors
• But, test scores in the system do not match hardcopy test scores in student files• The two cases that sealed his fate
Internal Process
• Notification of VPAA (February 7)
• More analysis (Weekend of February 8-9)• Audit of data entry changes shows one user affiliated with all changes on weekends in
September for numerous years (beginning in 2004)
• Update of VPAA (February 9 around 2am)
• Notification of President (February 10 at 8am)
• More analysis
• Final notification of President (February 12 at 8am)
• Trip with VPEM (February 13)
• Removal (February 14)
• Measure Impact (Weekend of February 15-16)
• Notify Cabinet on February 17 at 8am
Sampling of Findings
Sampling of Findings
Contributing Factors
• Relationships
• Whistle blower
• Access to databases
• Test score entry
• Policies on paper v. policies in practice
• Silos
The Public Process
• External investigators
• External data auditors
• Crisis communication consultants
• Openness and transparency• Stakeholders
• Campus community
Sampling of Findings
Sampling of Findings
Reporting and Response
• Open and transparent• Easier to tell all at the beginning
• No sugar coating
• Responses from community
• Responses from stakeholders• Accrediting bodies
• Government
• Publications
Impact
• Faculty
• Students
• Staff
• Institution as a whole
And the Other Shoe Falls
• Students aren’t impacted…if they take math and English first semester• Which some students don’t
• 223 students end up misplaced in math, English, or in a few cases both due to test score fraud
• A whole new process• Pull transcripts, courses taken, time of registration v. time of data
manipulation, determine remediation while consulting with faculty, presenting remediation, and in some cases scrub transcripts
• Interests DoE
Lessons Learned
Any press does not necessarily make for good press
Be open and transparent
Be selective in choosing investigators
Do not worry about cost…you’ll be remembered by how you respond
Lessons Learned
Do not sugarcoat
Policies exist to be followed
Do not hold press conferences
Some people don’t need access to some things
Lessons Learned
Do not blindly trust
Lessons Learned
It can be a positive impetus for change
• We’re actually a unicorn• Decreased applications, increased yield, made the class (and then
some), and increased profile
• Re-established relationships between enrollment management and faculty
• Personnel changes
Questions