PROGRAM OUTCOME ASSESSMENT: CASE STUDY FROM THREE ENGINEERING PROGRAMS
Presented by Toni L. Doolen, ProfessorSchool of Mechanical, Industrial & Manufacturing EngineeringOregon State University09/21/2011
AbstractThis presentation will introduce a framework for a continuous improvement process for programs driven by program outcome assessment. A case study illustrating the application of this framework to programs in the College of Engineering will be presented. Program outcome data collected from different sources (courses, students, and alumni) is being used to provide feedback to evaluate whether or not changes are needed in programs. A set of processes has been implemented to create an annual review cycle with a focus on continuous improvement of programs. Specific examples and templates for program outcome assessment and reporting will be provided.
Agenda Overview of the School of Mechanical,
Industrial, and Manufacturing Engineering (MIME)
Assessment and Evaluation in MIME Program Educational Objectives Student Outcomes Course Learning Outcomes
Continuous Improvement process
MIME Demographics
~35 full-time faculty members and instructors ~185 graduate students in three programs
(industrial engineering, material science mechanical engineering)
~1,375 undergraduates in five programs (energy engineering management, general, industrial, manufacturing, and mechanical)
~ $7.3 million in research funding in 2010
Need for Assessment and Evaluation
Undergraduate programs accredited under ABET
Many employers require accreditation Students must graduate from accredited
engineering program to sit for PE (Professional Engineering exam)
Three Levels of Assessment and Evaluation
Program
Educational
Objectives
(PEOs)
Student Outcomes
(SOs)Course Learning Outcomes
(CLOs)
Assessment Data collected from Multiple Stakeholders
Students Alumni Industrial Advisory Board Members Employers (local, regional, national,
international)
Levels of Assessment in MIME
MIME Program Educational Objectives (PEOs)
Student Outcomes
Course Learning Outcomes
Program Educational Objectives
Program educational objectives are broad statements that describe what graduates are expected to attain within a few years of graduation. Program educational objectives are based on the needs of the program’s constituencies.Program constituencies: students, faculty, employers/graduate schools, alumni, state of Oregon, …
PEOs A & E Process Defined for programs
Our graduates will be systems thinkers. Our graduates will be global collaborators. Our graduates will be innovative designers and
problem solvers. Assessed and evaluated using alumni (3 – 5
years after graduation) survey data, employer survey data, and IAB input.
Responsibility for A&E and CI process is MIME Head/AH, ABET coordinator, and undergraduate program committees
MIME Program Educational Objectives
•MIME graduates will be able to analyze, evaluate, improve, and design engineered systems and processes using modern engineering tools (hardware and software) and approaches. They will demonstrate in-depth knowledge of mechanical, industrial, and/or manufacturing systems.
Our graduates will be systems
thinkers.
•MIME graduates will be able to communicate effectively across disciplines and cultures. They will provide management and leadership skills within their organizations and work effectively in diverse environments.
Our graduates will be global collaborators.
•MIME graduates will use both structured and unstructured methodologies to innovate systems and processes. They will apply technical know-how and creativity to real-world problems. They will demonstrate a broad understanding of engineering design and problem-solving processes.
Our graduates will be innovative designers and
problem solvers.
PEO Assessment Data from Alumni Assessment of MIME PEOs is completed using an
annual Alumni Survey that is administered to alumni 2 – 5 years after graduation.
Items include specific questions about PEOs. Rated on a 7-point scale (1=not at all; 7=extremely). Respondents also rate importance of each PEO.
Survey provides an assessment of PEO achievement directly and also uses aggregated Student Outcome achievement ratings provided by alumni for a second measure.
Other demographic details are collected related to employment and general satisfaction with educational experiences.
PEO Evaluation PEOs and assessment results reviewed
with IAB annually. PEO data analysis and recommendations
prepared by ABET coordinator. Analysis and recommendations reviewed with UPCs on an annual basis and actions for improvement identified.
Linkage of PEO’s to Student Outcomes
Our graduates will be systems
thinkers.
•Ability to apply mathematics, science, and engineering•Ability to design and conduct experiments, as well as to analyze and interpret data•Ability to identify, formulate, and solve engineering problems•Ability to use the techniques, skills, and modern engineering tools necessary for engineering practice
Our graduates will be global
collaborators.•Ability to function on multi-disciplinary teams•Understanding of professional and ethical responsibility•Ability to communicate effectively•Knowledge of contemporary issues
Our graduates will be innovative designers and problem solvers.
•Ability to design a system, component, or process to meet desired needs within realistic constraints •Broad education necessary to understand the impact of engineering solutions in a global and societal context•Recognition of the need for, and an ability to engage in life-long learning
MIME Program Educational Objectives (PEOs)
Student Outcomes
Course Learning Outcomes
Student OutcomesStudent outcomes describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire as they progress through the program
Student Outcomes A&E Process General and Program-specific student
outcomes defined e.g. Assessed and evaluated using exit
surveys completed by students in the quarter they graduate and through Course Learning Outcome mapping
Responsibility for A&E and CI process is ABET coordinator and UPC
Student Outcomes Examples General outcomes for all programs
Ability to design and conduct experiments, as well as to analyze and interpret data
Ability to design a system, component, or process to meet desired needs
Ability to function on multi-disciplinary teams Program-specific outcomes
Ability to design, develop, implement and improve integrated systems that include people, materials, information, equipment, and people
Ability to work professionally in mechanical systems area including the design & realization of such systems
Student Outcome Assessment via Exit Survey Assessment of student outcomes is completed
using an annual Exit Survey that is administered to graduates in the term that they graduate, i.e. administered all 4 quarters.
Items cover Student Outcomes that are rated on a 7-point scale (1=not at all; 7=extremely).
Survey provides a direct assessment of program specific students outcomes in addition to a-k.
Other demographic details are collected related to coop experiences, number of interviews, employment status, and general satisfaction with their educational experiences (labs, faculty, advising, facilities, classes, etc.)
Student Outcome Evaluation Process Student Outcome data analysis and
recommendations are prepared by ABET coordinator. Analysis and recommendations reviewed with UPCs on an annual basis and actions for improvement identified.
MIME Program Educational Objectives (PEOs)
Student Outcomes
Course Learning Outcomes
Course-level Assessment and Evaluation
CLOs
Course Learning OutcomesCourse learning outcomes are statements that describe what students are expected to know and/or be able to do at the completion of a specific course.
CLOs 3 – 7 LO’s typical for a course Examples of LO’s
Be able to identify and describe (in writing) the most significant challenges faced by engineering managers in organizations in today's global environment.
Be able to identify and facilitate solutions to ethical dilemmas faced by engineering managers
Student Outcome Assessment via Course-Level Assessment and Evaluation Faculty must also identify two other direct measures to evaluate
the achievement of Course Learning Outcomes and ultimately of the linked Program Outcome. Specific assignments, exams, project reports, and grading rubrics are used by faculty for direct measures as appropriate to a given course.
Faculty members are responsible for collecting the necessary data for assessment and for completing appropriate analyses of the data to determine whether or not Program Outcomes have been achieved.
Faculty must also provide a narrative summary of their evaluation. This process is completed annually for all required courses (or every other year for courses taught on an every other year basis). Faculty members are responsible for reporting any significant issues in meeting Program Outcomes to the UPC. Evidence of this process is provided through the course notebooks.
CLO Assessment Plan Example
Evaluation of Learning Outcomes
CLO Evaluation Example
Expectation is for at least three distinct measures for each learning outcome, one of these measures can be student evaluation of CLO achievement for SET
Continuous Improvement Process
Continuous Improvement Cycle for all A&E Processes
Plan
DoStud
y
Act Review &
Update Desired
OutcomesA&E
Performance
against Outcome
s
Measure Impact
of Changes
Implement
Changes to
Address Gaps
PEO CI Process Examples
DoAct
Study
PEO CI Process Examples
Plan
SO CI Process
Study
PlanDo
Study
Act
CI Process: Plan
In response to the ABET reviewer’s concerns on thermal design and the persistent low achievement rating of Outcome c, the ME ABET coordinator as well as the UPC began discussions on possibility of enhancing the design experience for students in Fall 2009. A recommendation from the coordinator and the UPC on this matter to the ME faculty is tentatively scheduled for Fall 2010. Plan
CI Process: DoConsidering the low achievement rating on Outcome c and d, ME 382 was revised in its Fall 2009 offering to include a greater emphasis on designing real world products and forming diverse teams (using personality and skills as metrics). This change in project focus, while retaining the same design methodology topics, is expected to increase student awareness of the relevance of design to the mechanical engineering profession through is concrete experience with real mechanical systems (covering all domains of mechanical systems). Any potential impacts from these revisions will be first captured in the 2011 exit surveys (since ME382 is a junior-level course). Since Fall 2009, ME382 has been made a required course for MfgE and an elective for IE students. This ensures that the teams are more multi-disciplinary. Any potential impacts from these revisions will be first captured in the 2011 exit surveys.
Do
CI Process: Study and ActOutcome c on design, Fig. 2 indicates that this Outcome is rated low in comparison with importance as judged by Alumni. Also, performance on Outcome d seems unchanged despite the merger of ME and IE in senior design curriculum. It is recommended that the ME UPC focus on addressing the Outcomes c and d carefully over the coming years.
Study
Act
CLO CI ProcessExecutive Summary
Review of Previous Year In the Fall of 2005, substantial changes were made to the ENGR 111 course structure. Most notably, a transitional learning community concept was employed along with the addition of a significant number of active learning components. The focus for Fall of 2006 was to improve further upon these major structural changes by further adjusting the content and logistics with a focus on student retention and academic success. How the Concerns from Last Time Were Met This Time The most important change in the course, from a structure standpoint, in response to the concerns from Fall 2005, was to hire undergraduate TA’s who would be responsible for conducting both the laboratory and the TLC’s to help make sure that the content was more tightly integrated. In addition, course learning outcomes were modified to better reflect the goals and course purpose. Learning outcomes from three other introductory engineering courses were reviewed to help create the revised set of outcomes…………….. Results from This Term Course metrics used to evaluate the achievement of course learning objectives included homework assignments, two exams, a term project, and a student self-assessment. Each learning outcome, except Learning Outcome 3 (Compile a feasible program of study for your chosen engineering discipline) was measured using at least three distinct metrics. For the student self-assessment of learning gains, students were asked the following question: As a result of your work in this class, how well do you think that you now understand each of the following? ………. Recommendations for the Next Time This Course Is Taught Based on the numerical analysis of the metrics and a review of the student self-assessment of learning gains, the only Learning Outcome flagged for potential improvement was Learning Outcome 4. While the objective measures for this learning outcome were well above 40%, student self assessment averaged 2.83, corresponding on the qualitative scale between a little and somewhat. Eliminate Engineering Student Survival text book Find a single text book that couples student success skills training with the technical content Make homework more purposeful so students don’t perceive it as busy work (this should help improve
perceptions of gains related to Learning Outcome 4) Get guest lecturers to focus more on what they do within their actual day and projects Where possible show videos on the various engineering disciplines Add content to the labs to relate the laboratory exercise to the specific discipline
PlanDoStudyAct
