Development of processes and criteria for CEAB graduate attribute assessment Lynann Clapham...

Development of processes and criteria for CEAB graduate attribute assessment

Lynann ClaphamAssociate Dean (Academic)

Brian FrankDirector (Program Development)

Faculty of Applied ScienceQueen's University

FACULTY OF APPLIED SCIENCE

2

Overview

• Outcome requirements at CEAB and general higher education in Canada

• Nationwide engineering program responses• Outcomes measurement• Queen's University progress


3

CEAB accreditation changes

Engineering programs are being required to assess outcomes and produce a process for improvement, not just inputs

Section “3.1: Graduate attributes” (outcomes)


4

Section 3.1: Graduate Attributes

Two part requirement:

“The institution must (1) demonstrate that the graduates of a program possess the

attributes under the following headings. The attributes will be interpreted in the context of candidates at the time of graduation...

(2) There must be processes in place that demonstrate that program outcomes are being assessed in the context of these attributes, and that the results are applied to the further development of the program.”


5

Graduate attribute measurement

This requires:– Programs gather information that is measurable

and meaningful for them to use, not just to show CEAB accreditation teams

– Programs have a system to gather and analyze data over multiple years

– Programs gather enough information to improve the program, which may involve gathering information from students in first year, middle years, and graduating year of the program, though only graduating information is required


6

Why is CEAB requiring outcomesassessment?

1. Most major industrialized countries use them to demonstrate their students' capabilities, both in engineering and in other programs.

2. There was a danger that our students would not have their educational credentials recognized outside Canada if we cannot demonstrate outcomes.

Washington Accord: substantial equivalency among Australia, Canada, Hong Kong, Republic of Ireland, New Zealand, South Africa, United Kingdom, and United States, Japan, Singapore,Korea, and Chinese Taipei

Bologna Declaration: compatibility among higher education programs across Europe


7

Other forces: National standards

Provincial Ministers of Education across Canada released Ministerial Statement on Quality Assurance of Degree Education in Canada, which recommended outcome measures in for higher education

Ontario adopted the recommendations and released them as Undergraduate Degree Learning Expectations for all undergraduate programs in the province.

Similar expectations are likely to follow in other provinces.


8

Ontario requirements

Undergraduate Degree Level Expectations (UDLEs) based on national recommendations

1. Depth and Breadth of Knowledge 2. Knowledge of Methodologies 3. Application of Knowledge 4. Communication Skills 5. Awareness of Limits of Knowledge 6. Autonomy and Professional Capacity

Detailed expectations shown in document referenced in paper. They align with CEAB graduate attributes very well.


9

Why are graduate attributes (outcomes) being adopted?

It provides information about our students' knowledge, skills, and attitudes (outcomes), in addition to syllabi, instructor education, library resources, etc. (inputs).

It identifies gaps between our perceptions of what we teach and what students can do.

The information can be used to continually improve our programs.


10

Curriculum design

Currently most of us have designed curriculum around subject content:

ThermodynamicsFluid mechanicsElectric circuitsEconomicsSolid mechanics

This will require we consider what we want students to do with the content:

E.g. Students can:...select appropriate mathematical

principles and tools to analyze and solve engineering problems ... (Knowledge)

...evaluate and use information based on its authority, objectivity, and currency. (Lifelong learning)


11

What we have been given:

12 broad areas (graduate attributes), and a requirement to create a process for program improvement. Programs will need to decide:

Graduate attributes

How to measure?

In which courses?

What are specific criteria? (measurable)

Process for programimprovement?


12

Generally accepted process to do this

Identify stakeholders (advisory boards, faculty, students, staff, alumni)

Identify the program's major objectives (which must include CEAB graduate attributes!)

Create specific measurable assessment criteria for objectives

– e.g. “Students describe limitations of theory in predicting behaviour of processes and systems”

Identify assessment measures appropriate for criteria– e.g. report, oral presentation, simulation,

standardized test, lab performance, etc. Curriculum process flow by Peter Wolf, U Guelph


13

E.g. ABET's assessment flow chart


14

Creating assessment criteria

• What are a program's specific and measurable criteria?• Can involve cognitive (knowing, analyzing, creating),

beliefs, skills • Assessment criteria describes

1. level of expectation (“describes”, “compares”, “applies”, “creates”, etc.)

2. Content areae.g.

The student identifies constraints on a design problem including applicable standards, economic, environmental, culture, and societal considerations, health and safety risks


15

Example: Draft assessment criteria at Queen's University


16

Curriculum mapping: Link criteria toappropriate courses:

E.g. For a capstone course, a program might decide that the assessment criteria used are:

D1. identify constraints including applicable standards, economic, environmental, culture, and societal consideration, health and safety risks (design)

D2. use creativity to establish innovative candidate solutions. (design)L1. evaluate information from a variety of sources for objectivity,

authority, currency (lifelong learning)L2. describe opportunities for professional development in the field

(lifelong learning)...

These would be assessed using whatever is deemed appropriate: reports, presentations, portfolios, surveys, etc.


17

Graduate attributes similar to USA, UK, Australia, Korea, Peru,...

CEAB ABET UK-SPEC Eng Australia

Knowledge Knowledge Knowledge Knowledge

Problem Analysis Solve problems Eng. analysis Problem solving

Investigation Experiments

Design Design Design Design

Engineering tools Techniques, tools Lab skills Techniques, tools

Teamwork Teamwork Teamwork

Communication Communication Communication

Professionalism Contemporary issues Eng practice Professionalism

Impact Impact Social context Responsibilities

Ethics & Equity Ethical responsibility Ethical conduct Ethics

Econ. & Proj. Manage *(part of design) *(part of design)

Lifelong learning Lifelong learning Lifelong learning


18

Since expectations are similar, we can usematerial from other countries

General engineering outcomes– Learning Outcomes for UK-SPEC (UK)– Engineers Australia Competency Standards

(Australia)– EC2000 Outcome Attributes for ABET (USA)– ABET criteria 2009-2010– OCAV Undergraduate Degree-level expectations

(Ontario)– CDIO Syllabus

Available training: ABET's week long Institute for Development of Excellence in Assessment Leadership (IDEAL), run 2x/year


19

National response of deans

Similarity among Canadian programs (more so than among US institutions)

but

no large-scale funding to support development of assessment procedures (like NSF funding in US for EC2000)

So... how can we work together?


20

NCDEAS Education Committee

• National Council of Deans of Engineering and Applied Science (NCDEAS) Education Committee developing three “exemplars” to be shared among other schools as a starting point

– Queen's U (coordinating institution and clearinghouse)

– U Calgary– U Sherbrooke


21

Queen's Development plan

1. Measurable faculty wide assessment criteria, e.g.: Identify constraints including applicable standards, economic,

environmental, culture, and societal consideration, health and safety risks (Design)

2. Leveled criteria (recommended expectations in first, middle, and final year to track progression and recommend curriculum targets)

3. Assessment measures for criteria, e.g. design reports, observations, exams, surveys, oral presentations,

standardized instruments, etc.

4. Recommendations for software to gather and manage data5. Individual program customizations, map to courses


22

Progress

• January: B. Frank attended ABET's Institute for Development of Excellence in Assessment Leadership (IDEAL)

• February: began gathering functional requirements for software to support process

• March: formed 7 working groups to consider “what do we expect our students to be able to do when they graduate?” in the 12 areas required by CEAB

• April: worked with assessment specialist on framework for assessment criteria

• June: rolling out software, developing documentation, transitioning courses

• July: Rough draft of assessment criteria assembled


23

General comments

Not all courses need to assess graduate attributesCreate plan to assess some graduate attributes in each

academic term (not necessarily all attributes every year). E.g. assess design in year 1, 3, 6 of accrediation cycle

Select how many students are being sampled Triangulate: multiple sources of assessment for each

assessment criteriaUse a mix of direct measures (by directly observing student

skill and indirect measures (e.g. Student self-reports, alumni opions, exit surveys, etc.)


24

Queen's Timeline

Report

Fall '11Fall '09

Pilot faculty wide outcomes in some courses

Pilot software

Evaluate

Share

Spring '10

Department outcomes

Map to curriculum

Create process for program improvement

Fall '10

Pilot program outcomes

Gather data

Evaluate

Create faculty wide outcomes

Evaluate, select software

Spring '09


25

Material to be shared

• Queen's U:– Leveled assessment criteria,description of

sources, with links to UDLEs– Assessment measures– Example course mapping– Documentation for open-source software to

support gathering and reporting– Collection of references to support development

(competencies from other accreditation bodies, curriculum development resources, etc.)

• Resources from U Calgary and U Sherbrooke


26

Software support

Lesson from U.S.: don't need to pour $millions into assessment, gathering far too much data.

Over the summer we assessed learning management systems that will be able to store a variety of types of assessment measures and materials to simplify reporting and analyzing.

Best option for us was Moodle (open source, customizable, extendible, low initial cost), but similar functionality is available in Blackboard, DesireToLearn


27

Moodle support for graduate attribute assessment


28

Material being collected:Assessment wiki

Wiki link here.


29

Input/questions/comments

Lynann [email protected]

Brian [email protected]

Date post:	12-Jan-2016
Category:	Documents
Upload:	nancy-griffin
View:	218 times
Download:	3 times

Development of processes and criteria for CEAB graduate attribute assessment Lynann Clapham...

Documents