www.inquiry2improvement.com
How a Culture of Inquiry Can Help Your College
Move the Needle on Student Success
Dr. Rob JohnstoneTexas Pathways to Progress Institute
Dallas, TXSeptember 22, 2014
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Acknowledgements Much of the content in this presentation in conjunction with
national projects such as Completion by Design, the Aspen Prize for CC Excellence, and Bridging Research, Information & Cultures (BRIC)
Content has also been developed by and with a host of national partners, including:
• Community College Research Center (CCRC)• Jobs for the Future• JBL Associates• Public Agenda• The Research & Planning (RP) Group
Infographics were primarily designed by Greg Stoup
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Formalistic Doublespeak…
• Give your 1st reaction to the following list of words:• Accountability• Accreditation• Assessment• Continuous Improvement• Data-Driven Decision Making• Evaluation• Institutional Effectiveness• Key Performance Indicators (KPIs)• Learning Outcomes Assessment• Performance-based Funding• Program Review• Strategic planning
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Recapturing the Movement…
• Vivid imagery from previous slide• Faculty, Student Services folks & leaders have been beaten over the head with these phrases they are correct to analyze that they too often
haven’t led to authentic improvement and the more formal the process, often…
•There is good news for our improvement efforts – you can do all of the things captured by those words in more authentic and less formalistic ways
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Changing the Conversations…
• We have to make the conversation about things faculty, staff & administrators care about – students, their learning, and improving their outcomes and lives
• Not everybody will come along – but we don’t need everybody
• Org Change Thought: Red light / Yellow light / Green light
• People need to see their expertise acknowledged and integrated – and the effect of their efforts on outcomes
6www.inquiry2improvement.com
A Culture of Inquiry & Action
A RESOURCE for INSTITUTIONAL change
April 2014
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 7
An Applied Inquiry Framework for Student Completion (CBD)
Stage 1 – Explore how to improve outcomes
Stage 2 – Gather meaningful evidence
Stage 3 – Discuss evidence broadly
Stage 4 – Use evidence to inform change
Stage 5 – Measure the impact of change
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 8
What is a Culture of Inquiry?
Institutional capacity for supporting open, honest and collaborative dialog focused on strengthening the institution and the outcomes of its students.
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 9
Culture of Inquiry: Features● Widespread access to user-friendly
information on student outcomes● Encouraging more people to ask a wider
collection of questions and use their evidence and conclusions to enhance decision making
● Reflective and dynamic discussions across constituency groups
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 10
Culture of Inquiry: More Features
● Continuous feedback so adjustments can be made along the way and processes can be adapted
● A sense of ownership over improving student outcomes – not blaming the student
● Using insight generated from inquiry to act at varying levels of the institution to create the conditions to improve outcomes
11www.inquiry2improvement.com
Explore how to improve student outcomes
Focus inquiry on designing approaches that improve student outcomes
STAGE 1
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
How We Spend Our Time Matters…
When was the last time you sat in a standing committee meeting on your campus that used evidence to explore a key student progress, completion, labor market, learning or equity outcome for more than 20 minutes?
What types of questions do we spend most of our organizational resources answering?
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Malcolm Gladwell talks about the right question…
13
• Link to full video: http://www.ted.com/talks/malcolm_gladwell_on_spaghetti_sauce
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
The Right Question in the CC World… Placement Tests & Cut Scores
14
• What was the problem we were trying to solve?• I’d posit it was something like:
“Can we find a short instrument that will help us assign incoming students to various levels of math & English?” Or…
“How do we ensure higher levels of course success in transfer-level math & English courses?
• Are these the best questions? Why or why not?
Note: current system of placement tests may not even be the best solution for this question: LBCC / CCRC
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
What if we tried to solve…
15
• What placement process is the most predictive of transfer-level course success?
Or….• What is the optimal curricular structure to ensure that the greatest number of students pass transfer-level math / English courses with appropriate rigor?
Or….
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
More questions…
16
• What is the optimal math curriculum that produces computational learning outcomes that most students will need in the real world?
Or…• Is writing about literature the optimal way to teach students the writing skills they need in their general education courses? What about in the real world?
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 17
Organizing Question of Improvement Science:
What problem are you trying to solve?
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
An Example from AB Tech in Asheville, NC…
18
• We have to make sure our improvement efforts to trying to solve the right problem “Right” is slippery – the problem at hand may be
important, but it might detract us from a larger problem that is of much higher impact to improving student outcomes
• AB Tech’s “One Stop” Onboarding process Original problem they were trying to solve: “How
do we address the chaotic and disconnected nature of the onboarding of our new students?
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
One Stop Address the Chaotic Onboarding Problem, but…
19
• As the AB Tech folks reflected on their efforts, they realized they might unintentionally communicate to students that it’s possible in four hours or one day to: assess interests match interests to careers pick a program register for classes figure out financial stability issues be ready for opening day be prepared for any bump in the road that might occur
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Perhaps…
20
• A better question / problem to solve might be: “How do we ensure that students get the services they need – when they need them – as they move through their educational trajectory at our college?”
• Note that this still covers having a more streamlined onboarding experience – but recognizes that it sits in a larger context to be addressed
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
To sum up the starting line stage…
21
• More focus on asking the right question, and ensuring we know what problem we are trying to solve – actually less focus on the data per se
• Sometimes exploring the data can help you realize you’ve been asking the wrong question - The Right Pepsi vs. The Right Pepsi’s
• Traditional questions: enrollment, course success, material covered
• Emerging questions: improving outcomes - completion, progress, learning, labor market, equity
22www.inquiry2improvement.com
Gather meaningful evidence
Collect high-quality, meaningful evidence at the student support, classroom, program,
and institutional levels
STAGE 2
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 23
When gathering evidence, make sure you are focusing on the right data…
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 24
Persistence Rate
Success Rate
1989 2008
20 year trend for California CCcourse success & persistence rates
What does that tell us about the usefulness of these metrics in setting institutional strategies?
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
The Aspen Prize’s Take on Data & Outcomes that Matter
• Progress / Completion Outcomes• Labor Market Outcomes• Learning Outcomes• Equity in Achieving First Three Outcomes
• Examples of each in Appendix presentation…
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Gather meaningful evidence for the “Four Big Ideas for Redesign”
1. Structured Pathways / Programs of Study
2. Better On-ramps into POS Accelerated / co-curricular Dev Ed, First Year Experience
3. Intentional Student Support Structures Intake system, onboarding, advising
4. Monitoring Progress / Customized Feedback
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 27
Progress Example 1:Grades in Sequenced Courses
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Grades in Sequenced Courses Backdrop
• Course sequences analyzed in Mathematics, English, Biology, Chemistry, Physics & Economics
• Relationship between grade in 1st course in sequence and success in 2nd course in sequence was examined.
• Enrollment data tracked over five-year period• Selected highlight slides follow; full PPT
provided on request
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Data Too Often Presented Like This…
Status in Math 310 Count% of Total
NSuccess Rate in Math 310
A in Math 308 5,659 10% 81%
B in Math 308 7,726 13% 60%
C in Math 308 7,234 12% 37%
Unclear Status in Math 308* 1,675 3% 16%
Math 310 Repeater / Non Success 14,568 25% 41%
Math 310 Attempt was First Math Course
16,812 29% 57%
Taking Math 308 / 310 in Same Semester
4,505 8% 68%
Other 487 1% 44%
Total 58,666 100% 53%
29
A
B
C
The grade received by students in Math 308
Pre-Algebra Introductory Algebra
Intermediate Algebra
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Success Rate of those same students in Math 310
37%
60%
81%
Math 306 Math 308 Math 310
Intermediate AlgebraMath 310
B
A
Transfer-Level
Those placing directly into
Math 31057%
Math 1300s
And Not Often Enough Like This…
A
B
C
The grade received by students in Math 2413
Calculus I Calculus II
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Success Rate of those same students in Math 2414
50%
78%
90%
Calculus Course Sequence
MATH 2413 MATH 2414
Calculus II
MATH 2414
B
A Those placing directly into Math
2414
72%
A
B
C
The grade received by students in English 307
One Level below
Transfer
One Level below
Transfer
Two Levels below
Transfer
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Success Rate of those same students in English 1301
46%
61%
74%
Developmental Education Writing Course Sequence
Those placing directly into English 1301
ENGL 307ENGL 306
Transfer Level English
ENGL 1301
B
A71%
A
B
C
The grade received by students in Biology 2401
Anatomy Physiology
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Success Rate of those same students in Biology 2402
50%
72%
88%
College Biology Course Sequence
BIO 2401 BIO 2402
Physiology
BIO 2402
B
AThose placing directly into Biology 2402
67%
A
B
C
The grade received by students in Chemistry 1411
General Chemistry I
General Chemistry II
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Success Rate of those same students in Chemistry 1412
46%
70%
86%
College Chemistry Course Sequence
CHEM 1411 CHEM 1412
General Chemistry II
CHEM 1412
B
AThose placing directly into
Chemistry 141266%
A
B
C
The grade received by students in Economics 2301
Macro Economics
Micro Economics
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Success Rate of those same students in Economics 2302
74%81%
89%
Economics Course Sequence
ECON 2301 ECON 2302
Micro Economics
ECON 2302
B
A Those placing directly into
Economics 230271%
36www.inquiry2improvement.com
Discuss evidence broadly
Engage a variety of campus stakeholders in evidence-based discussions about
improvements in practice
STAGE 3
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Key Concept: Data do not speak for themselves
Time & Space Needed to Explore Data, Make Meaning & Generate
Insight
37
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 38
Exploring Data Example: English & Math Preparedness
& Success in GE Courses
Not in an English course
Engl 826
Engl 836
Engl 100
Engl 110
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
31%
48%
68%
75%Not in an English course
64%
67% 33%
Success Rate of those same students in Psychology 101
English course taking profile for students in Psychology 101
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
Success in Psychology 101 for students simultaneously enrolled in an English course
English 826(Two Below)
English 836(One Below)
English 100(Transfer A)
English 110(Transfer B)
Not in a Math course
Funda-mentals
Beginning Algebra
Intermediate Algebra
Transfer Level
0%10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
43%
51%
63%
76%Not in an Math course
64%
63% 37%
Success Rate of those same students in Psychology 101
Math course taking profile for students in Psychology 101
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
Success in Psychology 101 for students simultaneously enrolled in an Math course
Fundamentals Beginning Algebra
Intermediate Algebra
Transfer Level Math
0%10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
31%
48%
68%
75%
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
Success in five highly enrolled GE courses by English enrollment level
Psychology 101 Speech 101 Economics 101 History 101 HSCI 101
54% 57%
73%
82%
27%
56%
73%79%
48%43%
51%
64%60%
74%
82%88%
English 826(Two Below)
English 836(One Below)
English 100(Transfer A)
English 110(Transfer B)
0%10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
48%
58%
68%
76%
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
Success in five highly enrolled GE courses by English enrollment level
Fitness 334 Accounting 101 Music 202 Biology 250 Sociology 101
37%
50%
66%
73%
31%
64%
74%80%
40%
60%65%
50%
59%
74%
English 826(Two Below)
English 836(One Below)
English 100(Transfer A)
English 110(Transfer B)
40%
N/A
www.inquiry2improvement.com
Use evidence to inform change
43
Implement changes in practice and policy based
on analyses and discussion of college evidence
STAGE 4
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement
Use evidence to guide innovation
● In this context, research and applied inquiry are fundamentally interventionist in nature.
● We are not seeking absolute truths; rather we are looking for patterns of evidence that inform action-oriented decisions.
● Failure can be seen as an opportunity for learning, especially when outcomes are shared and used to inform further improvements in practice.
44
The process of inquiry is not a search for an absolute truth
Domain of possible solutions
We answer the questions that eliminate dead end solutions
And what do we do when the evidence is ambiguous?
Gregory M Stoup, Cañada College
What to do when you reach the limits of your research and yet still face multiple choices?
Trust your expertise & choose !
www.inquiry2improvement.com
Measure the impact of change
46
Evaluate the impact of practice changes on
student outcomes
STAGE 5
www.inquiry2improvement.com
Final Thoughts
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 48
Final Reflections
• Creating or evolving your culture of inquiry isn’t magic; there are clear steps and resulting artefacts of such a culture
• Don’t focus too much on the data – the questions you ask are exponentially more important
• Remind yourself and your team to constantly ask: “What problem are we trying to solve?”
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 49
What we are shooting for…
• Great statement of the desired end state from a CC President at the Aspen / ATD Leadership Symposium:
“A wider range of people on a campus ask a better set of questions about outcomes & act on their reflections to improve
them.”
www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 50
Find Out More
• The National Center for Inquiry & Improvement websitewww.inquiry2improvement.com
• Dr. Rob Johnstone, Founder & [email protected]
• CBD Inquiry Guides on Applied Inquiry & Nuances of Completion: http://www.inquiry2improvement.com/publications-resources