A Rubric-based Approach for Operationalizing the
Entrepreneurial Mindset: An exploratory investigation
J-D YoderProfessor and ChairMechanical Engineering
Rob KleineAssociate Professor of MarketingPharmaceutical Business Program Coordinator
We’ve got rubrics, and we’re here to help you.
Ohio Northern University• Approximately 3500 students among five
colleges• College of Engineering is about 450
students across 5 majors• Largely traditional, residential students
Kern Entrepreneurship Education Network (KEEN)
• A program of the Kern Family Foundation
• Started in 2005• Private universities with ABET
accreditation• Focus on the entrepreneurial
mindset, not entrepreneurship
The KEEN Network
Baylor University (TX)Boston University (MA)Bucknell University (NY)Gonzaga University (WA)Kettering University (MI)Lawrence Technological University (MI)Mercer University (GA)Milwaukee School of Engineering (WI) Ohio Northern University (OH) St. Louis University (MO)
Santa Clara University (CA)Union College (NY)University of Dayton (OH)University of Detroit-Mercy (MI)University of Evansville (IN)Villanova University (PA)Western New England University (MA)Widener University (PA)Worcester Polytechnic Institute (MA)
The Entrepreneurial Mindset (KEEN)1. Effectively collaborate in a team setting2. Apply critical & creative thinking to ambiguous
problems3. Construct & effectively communicate a customer-
appropriate value proposition4. Persist through and learn from failure5. Effectively manage projects through appropriate
commercialization or final delivery process6. Demonstrate voluntary social responsibility7. Relate personal liberties and free enterprise to
entrepreneurship
Motivation
• Many schools in the network have created engineering-mindset-related activities.
• How do we know which ones work?• How can we leverage “best in class”
activities?• Note – don’t want it to be relative to student
feelings about the activity, but on the outcomes.
Can you answer these questions?
• How effectively is my PROGRAM cultivating the entrepreneurial mindset in our students?
• How effective is each ELEMENT in my program at cultivating intended aspects of the entrepreneurial mindset (whatever the unit: minor, course, project, in-class activity, etc.)
Our effort:
• Develop a common set of evaluation tools.
• Rubric-based.• Evaluate student work.
A Rubric-based Approach for Operationalizing and Assessing the Entrepreneurial Mindset
Source: Kleine & Yoder (2011), JEEN
The process: Activity
StudentArtifact
RubricsData Analysis
ReflectionRevision?
Exploratory Investigations Utilizing the Rubric-Based Approach
• Faculty training protocol• Faculty using and testing the
rubrics
A faculty training protocol for rubric-based assessment Preparation (Kleine & Reid 2012)
1. A student learning outcome of interest2. A Rubric3. A student activity that generates student
work (artifacts) relevant to the learning outcome
4. At least two examples of student work generated by the activity:
– High proficiency – Low proficiency
Materials distributed to each participant
1. The rubric that will be applied. 2. A score sheet for each artifact to be
evaluated.3. Details about the activity (assignment)
that generated the student work. 4. Student work examples (artifacts) that
illustrate high and low proficiency (each on a separate page.
Example Artifact Score SheetArtifact Score Sheet
(Please Refer to the Effective Collaboration Rubric)
Does Not Meet Expectations (1) Developing (2)
Meets Expectations (3) Proficient (4)
Not Applicable
Contributes to Team Meetings
Facilitates the Contributions of Team Members
Fosters Constructive Team Climate
Responds to Conflict
Activity: A Reflection Paper A one-page “Team reflection” should be written to describe the results of your “Team evaluation” rubric. This memo should allow you to reflect on whether your team has been effective, especially considering qualities of effective teams we have discussed. You may also include any concerns about team dynamics (these concerns will not be shared with your teammates without your consent).
Sample Artifact Low Proficiency (Student JJ)Team LL has had a rough start. After having six team meetings the group has failed to have a hundred percent attendance to a single meeting (AA has missed 4, DD 3, RR 1, and JJ 0.) Without the whole group present, our decision making and brainstorming has been continuously pushed back. When we do have meetings they are somewhat productive, however, for the following meeting we must repeat what was said previously to ensure everyone is on the same page. Our team meetings usually last one hour, during which I do most of the talking while everyone else just agrees with me. Being the team leader I understand that it is my job to run the meetings and to keep everyone on track, but it becomes a nuisance when no one else has any suggestions and must be told to help. The poor attendance is a result of poor communication skills. Now that email addresses and phone numbers were exchanged I hope that communicating will no longer be a problem.
Sample Artifact Low Proficiency (Student JJ)After having six team meetings … When we do have meetings they are somewhat productive, …
… Our team meetings … during which I do most of the talking while everyone else just agrees with me. Being the team leader I understand that it is my job to run the meetings and to keep everyone on track, but it becomes a nuisance when no one else has any suggestions and must be told to help.
The poor attendance is a result of poor communication skills. Now that email addresses and phone numbers were exchanged I hope that communicating will no longer be a problem.
Sample Artifact High Proficiency (Student MM)PW Inc. is made up of four members: NN, KK, MM, and SS. The team has worked well together and
all the members get along with each other very well. All the individuals on the Team have contributed to the project equally so far. This is very beneficial for the team because all the assignments are getting done in a reasonable amount of time. The culture identity was divided up so that each person had to research certain information. As a team everything has been evaluated and everyone has come to a consensus on team assignments so that no individual’s opinion is less important than another. Each person has pulled his or her weight on assignments and/or a task assigned to them, which shows that everyone has respect for each other and wants to succeed. The key characteristic of the PW Inc. team is our ability to effectively communicate. The team has a lot of meetings to insure that everything can get done in a timely manner, so communication is key and so far it has been excellent. In addition to communication, attendance at each team meeting is valuable to getting things accomplished in a timely manner which the group has done a great job of so far. The team doesn’t waste time on unimportant things, but rather focuses on the duties that need to get done, such as deciding the company name, working on the cultural identity, and creating the company.
One aspect I feel like the team has not done well is in group dynamics. I believe that some group members don’t always participate in discussions because he or she doesn’t want to be wrong or cause a distraction. My feeling is we are mature enough to not get offended if someone else has a better idea or an opinion that we don’t necessarily agree with at the time.
Working together and communication are the two main keys to a successful team and so far I believe that PW Inc. is building for success. We have followed these two keys and have accomplished all the tasks assigned to the group.
Sample Artifact High Proficiency (Student MM)As a team everything has been evaluated and everyone has come to a consensus … Each person has pulled his or her weight on assignments and/or a task assigned to them, which shows that everyone has respect for each other and wants to succeed. The key characteristic of the PW Inc. team is our ability to effectively communicate.
… communication is key and so far it has been excellent.
… The team doesn’t waste time on unimportant things, but rather focuses on the duties that need to get done, …
… the team has not done well is in group dynamics. I believe that some group members don’t always participate in discussions because he or she doesn’t want to be wrong or cause a distraction. My feeling is we are mature enough to not get offended if someone else has a better idea or an opinion that we don’t necessarily agree with at the time.
Working together and communication are the two main keys to a successful team … I believe that PW Inc. is building for success.
Example Artifact Score SheetArtifact Score Sheet
(Please Refer to the Effective Collaboration Rubric)
Does Not Meet Expectations (1) Developing (2)
Meets Expectations (3) Proficient (4)
Not Applicable
Contributes to Team Meetings
Facilitates the Contributions of Team Members
Fosters Constructive Team Climate
Responds to Conflict
Distribution of Ratings on the “Fosters Constructive Team Climate” row of the “Effective Collaboration in a Team Setting” rubric
Does Not M
eet Exp
ectations
Developing
Meets
Expecta
tions
Proficie
nt
Not Applic
able02468
1012
JJ MM
NOTE: n=23. The distributions are statistically distinct (Chi-squared=27.5, df=4, p<.001).
“Effective Collaboration in a Team Setting” Aggregate Ratings
Does Not M
eet Exp
ectations
Developing
Meets
Expecta
tions
Proficie
nt
Not Applic
able05
10152025303540
JJ MM
Faculty Using and Testing the Rubrics
• ONU Faculty in Colleges of Engineering and Business
• Faculty at other KEEN institutions
Rubric Testing Protocol
Pick a KEEN learning outcome
(or two)
Identify a class activity you do
already (or create a new one)
Apply the rubric(s) to one or more examples of
student work generated by the
activity
Provide feedback on how we can
improve the rubric(s)
Work in progress
• Intend to use in additional courses at ONU.
• Expand to other KEEN schools.• All schools are welcome to participate.• Use feedback to modify rubrics as
needed.• Inventory entrepreneurial mindset
activities by learning outcome
Mapping Activities into Entrepreneurial Mindset Student Outcomes
Feb. 2012 KEEN Student Outcomes (KSO) Contact: ___________________________
1 2 3 4 5 6 7
Effectively collaborate in a team setting
Apply critical & creative thinking to ambiguous problems
Construct & effectively communicate a customer-appropriate value proposition
Persist through and learn from failure
Effectively manage projects through appropriate commercialization or final delivery process
Demonstrate voluntary social responsibility
Relate personal liberties and free enterprise to entrepreneurship
For-Credit Courses Provide course number and title
Co-curricular Activities
Extra-curricular Activities
Opportunities
• Data Aggregation• Integrating with ABET• Integrating Entrepreneurial Mindset and
General Education assessment systems
Conclusion
Our goal is to answer the question empirically:
How effective are our efforts to cultivate the Entrepreneurial Mindset in our students at each of the following levels:
• KEEN network • Individual KEEN school • Program • Course • Assignment/activity
Thank You!
Questions?
Misc debris follows
Connect General Education and the Entrepreneurial MindsetONU Gen Ed Learning Outcomes1. Effective communication 2. Critical and creative thinking3. Scientific and quantitative literacy 4. An understanding of diverse cultures
and their effects on human interaction5. Integration of concepts across
disciplines6. Informed and ethical responses to
personal, civic, and global needs7. Informed responses to aesthetics in
art or nature
KEEN Learning Outcomes1. Effectively collaborate in a team setting2. Apply critical & creative thinking to
ambiguous problems3. Construct & effectively communicate a
customer-appropriate value proposition
4. Persist through and learn from failure. 5. Effectively manage projects through
appropriate commercialization or final delivery process
6. Demonstrate voluntary social responsibility
7. Relate personal liberties and free enterprise to entrepreneurship
Motivating vision
• Cultivate the entrepreneurial mindset in every ONU student.
• Support those students interested in becoming entrepreneurs
Entrepreneurial Mindset for All
Supporting Future Entrepreneurs
Zach Ferres
Chris Corrado
Entrepreneurship Education at ONU Circa 2004
Entrepreneurship education at ONU Circa 2004
A Multi-Pronged Approach to Engaging the Entrepreneurial Mindset
Curriculum
Faculty
Development
Extra/Co-Curricular
Activities
Assessment
Entrepreneurial Mindset Touch
Points across the Four Years
Highlights: Student Engagement
Polar Innovation Exchange Competitions
Student competitions
• Polar Innovation Exchange
• Elsewhere• IIT Idea Chase • Innovation Chase at LTU
Highlights: Speakers that Inspire the Entire Campus Community
Paul Polak author of Out of Poverty and co-founder International Development Enterprises
Bob Peterson, Pixar Studios
Entrepreneurship education at ONU Circa 2004
Thanks
• The Deans• Dr. Baker and Dr. DiBiasio• A variety of faculty, especially Dan
Ferguson and Tammy Schakett• Fantastic students• KEEN and NCIIA• External allies