Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | bryan-blair |
View: | 215 times |
Download: | 0 times |
Program development process at Queen’s University to demonstrate graduate attributes
Brian Frank
Director (Program Development)Faculty of Engineering and Applied Science
Queen's University
1
Focus
“The institution must demonstrate that the graduates of a program possess the attributes under the following headings... There must be processes in place that demonstrate that program outcomes are being assessed in the context of these attributes, and that the results are applied to the further development of the program.”
2
CEAB InstructionsDescribe the processes that are being or are planned to be used. This must include:a) a set of indicators that describe specific abilities expected of
students to demonstrate each attributeb) where attributes are developed and assessed within the
program…c) how the indicators were or will be assessed. This could be
based on assessment tools that include, but are not limited to, reports, oral presentations, …
d) evaluation of the data collected including analysis of student performance relative to program expectations
e) discussion of how the results will be used to further develop the program
f) a description of the ongoing process used by the program to assess and develop the program as described in (a)-(e) above
3Engineering Graduate Attribute Development (EGAD) Project
Approach• Short term objectives (2010-2011):
• Set up a comprehensive process limited to a small number of courses to help programs understand the process
• Use data to help faculty see value in outcomes assessment for program improvement
• Long term: • Comprehensive assessment of all attributes
throughout programs• Evaluate validity of data• Students take responsibility for demonstrating
some attributes
Queen's University timeline Summer 2009: Working groups of faculty, students,
topical experts created specific program-wide indicators (next slide, and in Appendix 3.1A)
Summer 2009: Setup learning management system (Moodle) to manage assessments
Sept 2009-April 2010: Piloted assessment in first year
Sept 2010-April 2011: Piloted assessment in first year, faculty wide second year, and fourth year (common across programs)
April – July 2011: Student surveys and focus groups, curriculum mapping, data analysis
Curriculum planning happening throughout
Why initial emphasis on first year?
• First year is faculty-delivered, core to all students
• Provides opportunity to pilot a process• Help disseminate outcomes
assessment procedures to other instructors
• Long term: assessment process continue in first year program to inform development
Aside: Idealistic course development process
7Engineering Graduate Attribute Development (EGAD) Project
Identify courseobjectives and
content
Create specific outcomes for each
class
Map to experiences(lectures, projects,
labs, etc.)
Identify appropriatetools to assess
(reports, simulation,tests,...)
Student input
Deliver, grade, seek feedback
Analyze and evaluate data
OverallImprovement
Create and Execute a Plan
Program-wide assessment process flow
8Engineering Graduate Attribute Development (EGAD) Project
Defining Purpose and Outcomes
ProgramMapping
Stakeholder input
Identifying and Collecting Data
Analysis andInterpretation
Create a ProgramImprovement Plan
Program & CourseImprovement
Human capital
• Director, Program Development to manage process
• Faculty member from each program• Other experts as appropriate
(economics, information management, etc.)
Currently separate from faculty-wide curriculum development committee
9
Resources/time commitment Creating assessment criteria: 7 committees
of approximately 5 people who each met about 4 times
Mapping criteria to a course and creating rubrics for assessment: ~ 10 hours
Large scale curricular changes: ~10 person committee, most of whom had 1 course relief bought out by dean
Coordination (resource gathering, planning, curricular planning): ~30% of a position
Academic and curricular structure
Dean
Associate Dean (Academic)
Director (Program Development)
NSERC Design Chair
DuPont Canada Chair in Engineering Education
Faculty-wide curriculum committee
Dean’s Retreat Curriculum Review Committee(DRCRC)
Graduate attribute assessment committee
What are indicators?
Engineering Graduate Attribute Development (EGAD) Project 12
Lifelong learningAn ability to identify and address their own educational needs in a changingworld in ways sufficient to maintain their competence and to allow them to
contribute to the advancement of knowledge
Can this be directly measured?
Would multiple assessorsbe consistent?
How meaningful would the assessment be?
Probably not, so more specific measurable indicators are needed.This allows the program to decide what is important
Indicators: examples
Engineering Graduate Attribute Development (EGAD) Project 13
Lifelong learningAn ability to identify and address their own educational needs in a changingworld in ways sufficient to maintain their competence and to allow them to
contribute to the advancement of knowledge
Critically evaluates informationfor authority, currency, and
objectivity when referencingliterature.
Uses information ethically and legally to accomplish a specific purpose
Identify gap in knowledge and develop a plan to address
Graduateattribute
The student:
Describes the types of literature of their field and how it is produced
Indicators
Establishing Indicators
• A well-written indicator includes:• what students will do• the level of complexity at which they will do it• the conditions under which the learning will
be demonstratedEngineering Graduate Attribute Development (EGAD) Project
14
Critically evaluates information for authority, currency, and objectivity in reports.
Content areaLevel of expectation(“describes”, “compares”, “applies”, “creates”, etc.)
context
Graduate attribute
categories
levels
Assessment criteria
15Engineering Graduate Attribute Development (EGAD) Project
Linkage to OCAV UDLEs
Rubric example
Creating defined levels (“scales”) of expectations reduces variability between graders, makes expectations clear to students
threshold target
Sample First year indicators for problem analysis and design
17Engineering Graduate Attribute Development (EGAD) Project
3.02-FY1 Identifies known and unknown information, uncertainties, and biases when presented a complex ill-structured problem
3.02-FY2 Creates process for solving problem including justified approximations and assumptions3.02-FY3 Selects and applies appropriate quantitative model and analysis to solve problems3.02-FY4 Evaluates validity of results and model for error, uncertainty3.03-FY1 Generates ideas and working hypothesis3.03-FY2 Designs investigations involving information and data gathering, analysis, and/or
experimentation3.03-FY3 Synthesizes data and information to reach conclusion3.03-FY4 Appraises the validity of conclusion relative to the degrees of error and limitations of theory
and measurement3.04-FY1 Adapts general design process to design system, component, or process to solve open-
ended complex problem.3.04-FY2 Accurately identifies significance and nature of a complex, open-ended problem3.04-FY3 Identifies customer and user needs3.04-FY4 Gathers and uses information from appropriate sources, including applicable standards,
patents, regulations as appropriate.3.04-FY5 Produces a variety of potential design solutions suited to meet functional specifications3.04-FY6 Performs systematic evaluations of the degree to which several design concept options
meet project criteria3.04-FY7 Compares the design solution against the problem objective
Sample fourth year indicators for Problem analysis and Design
18Engineering Graduate Attribute Development (EGAD) Project
3.02-GY1 Identifies problem, known and unknown information, uncertainties, and biases
3.02-GY2 Creates process for solving problem including justified approximations and assumptions
3.02-GY3 Selects and applies appropriate model and analysis to solve problems
3.02-GY4 Evaluates validity of results and model for error, uncertainty
3.04-GY1 Identify problem and constraints including health and safety risks, applicable standards, economic, environmental, cultural and societal considerations
3.04-GY2 Applies appropriate knowledge, judgement, and design tools, in creating and analyzing conceptual design solutions to select best concept
3.04-GY3 Creates and tests simulations, models, and/or prototypes at various points in design with complexity appropriate to design stage
3.04-GY4 Assesses design performance based on requirements, yield, reliability, and/or safety as appropriate
3.04-GY5 Identifies possibilities for further improvement and conducts design review to evaluate performance of the overall process.
Program-wide assessment process flow
19Engineering Graduate Attribute Development (EGAD) Project
Defining Purpose and Outcomes
ProgramMapping
Stakeholder input
Identifying and Collecting Data
Analysis andInterpretation
Create a ProgramImprovement Plan
Program & CourseImprovement
A knowledge base for e
ngineering
Communication sk
ills
Design
Economics
and project
management
Ethics and equity
Impact
of engineerin
g on socie
ty and the enviro
nment
Individual and te
am work
Investigation
Life-lo
ng learn
ing
Problem analysis
Professi
onalism
Use of engineerin
g tools
0
50
100
150
200
250
Development of Enginering Attributes
Not Taught / Not Assessed
Not Taught / Assessed
Taught / Not Assessed
Taught / Assessed
Student surveys and focus groups
• Provides student input:• implementing attribute assessment in
program• perceptions on where attributes are
developed within the program as complement to curriculum mapping via faculty survey
• perception of importance within program
Questions
• What do you think are priorities within the program?
• What courses contribute to development of attribute {}?
• Which attributes are difficult to demonstrate?
• How would you recommend that attributes be developed?
Self reported demonstration at program entry
Top five Grad Attributes where students reported a rating of 2 or 3 (yes or to a great degree) out of 3 Individual and Team Work 88.73% Communication Skills 78.17% Professionalism 69.02% Problem Analysis 61.26% Investigation 60.56%
Potential for students to perceive little value in learning activities directed toward developing these attributes
First year program supports:
27
Attributes in students’ top five responses Individual and Team Work* 94.97% Knowledge Base in Engineering 93.53% Problem Analysis* 93.53% Professionalism* 85.58% Investigation* 82.48% Design 80.58% Impact of Engineering on Society 80.58% *Identified as a strength coming in to the program
First year program supports
28
Bottom three responses Ethics and Equity 64.03% Economics and Project Management 69.56% Lifelong Learning 73.19%
These three are a significant focus in APSC-100, embedded in various activities.
Attributes perceived to be program priorities
29
Attribute # of students who incl it in top 5
# of students who selected it as #1 priority
Problem Solving* 120 34
Individual & Team Work* 97 17
Knowledge Base 95 51
Communication* 76 6
Professionalism* 47 4
Graduating students: low priority attributes in program
30
Attribute # who included it in bottom 3
# who ranked it lowest priority
Lifelong learning 80 38
Economics and Project
Management
68 25
Ethics and Equity 45 1
Use of Eng Tools 38 15
Impact on Society
37 6
Focus group suggestions• Communicate graduate attributes and draw
attention back to them• What is lifelong learning”?• Professionalism and ethics and equity should be
focused on in upper years
Program-wide assessment process flow
32Engineering Graduate Attribute Development (EGAD) Project
Defining Purpose and Outcomes
ProgramMapping
Stakeholder input
Identifying and Collecting Data
Analysis andInterpretation
Create a ProgramImprovement Plan
Program & CourseImprovement
Assessment in 2010-2011Attribute First year Upper yearsKnowledge base FCI, CalculusProblem analysis Complex problem
solvingCapstones
InvestigationDesign Design project CapstonesEngineering tools Complex problem
solvingCapstones
Communications Design project Communications, Capstones
Individual and teamwork
Design project Capstones
Impact of engineering Design project CapstonesProfessionalism Complex problem
solvingCapstones
Ethics and equity Complex problem solving
Capstones
Economics EconomicsLifelong learning Design project Capstones 33
Analyze and evaluate… Histogram of results by level (did or
did not meet expectations) Histogram of results by student (how
many indicators did each student fall below
Trend over time Triangulation: examination of
correlation between results on multiple assessments of the same indicator data with exam results)
Threshold Target 1 - Not Demonstrated 2 - Marginal 3 - Meets Expectations 4 - Outstanding
3.02 - FY1: Identifies known and unknown information, uncertainties, and biases when presented a complex ill-structured problem
Information not identified properly, no information, or information copied from assignment
Some important information or biases not identified, or trivial/incorrect information included
Identifies known and unknown information, uncertainties, and biases
Meets expectations PLUS: Includes information from authoritative sources to inform process, model, and conclusions
3.02 - FY2: Creates process for solving problem including justified approximations and assumptions
No or inadequate process
Process identified misses some important factors; some assumptions left unidentified or unjustified.
Creates justified process for solving problem, suppored by information.
Meets expectations PLUS: Comprehensive process model; comparison with other possible approaches
3.02 - FY3: Selects and applies appropriate quantitative model and analysis to solve problems
No analysis, or model/analysis selected is inappropriate
Model selected; some errors in analysis or inappropriate assumptions
Selects and applies approriate quantitative model and MATLAB analysis to solve problems, using reasonable approximations and assumptions
Meets expectations PLUS: Authoritative research used to defend assumptions and approximations made
3.02 - FY4: Evaluates validity of results and model for error, uncertainty
No evaluation of solution
Superficial evaluation of solution
Evaluates validity of results and model for error, uncertainty
Meets expectations PLUS: Evaluates conclusions and presents potential improvements
Threshold Target 1 - Not Demonstrated 2 - Marginal 3 - Meets Expectations 4 - Outstanding
3.07 - FY3: Summarizes and paraphrases written work accurately with appropriate citations
Insuffi cient content to assess summary of work. Summary misinterprets researched material.
Records information from few resources. Misses significant points of view.
Summarizes and paraphrases written work accurately.
Synthesizes main ideas to construct new concepts. Summarizes the leading thoughts in the field and gives a broader picture of the problem.
FEA
S - 3
.07-
FY3
(Pro
pRep
ortP
M)
FEA
S - 3
.07-
FY3
(Pro
pRep
ortF
S)
FEA
S - 3
.07-
FY3
(Pro
pRep
ortA
A)
FEA
S - 3
.07-
FY3
(T2S
cope
&In
fo)
APSC100 APSC100 APSC100 APSC100
0100200300400
1234
time
Knowledge base: MathematicsCalculus instructor asked questions on exam that specifically targeted 3 indicators for “Knowledge”:
1. “Create mathematical descriptions or expressions to model a real-world problem”
2. “Select and describe appropriate tools to solve mathematical problems that arise from modeling a real-world problem”
3. “Use solution to mathematical problems to inform the real-world problem that gave rise to it”
Indicator 1:
• The student can create and/or select mathematical descriptions or expressions for simple real-world problems involving rates of change and processes of accumulation (overlaps problem analysis)
39
Context: calculatingIntersection of two trajectories
Indicator 2:Students can select and describe appropriate tools to solve the mathematical problems that arise from this analysis
40
Context: differentiationsimilar to high school curriculum
Indicator 2:• Students can select and describe
appropriate tools to solve the mathematical problems that arise from this analysis
41
Context: implicit differentiation, triginverse
Program-wide assessment process flow
42Engineering Graduate Attribute Development (EGAD) Project
Defining Purpose and Outcomes
ProgramMapping
Stakeholder input
Collecting Data
Analysis andInterpretation
Create a ProgramImprovement Plan
Program & CourseImprovement
2010-08 2010-09 2010-11 2011-01 2011-02 2011-04
10
20
30
40
50
2.000
2.200
2.400
2.600
2.800
3.000
3.200
3.400
3.600
3.800
4.000
% Below targetLinear (% Below target)MeanLinear (Mean)
Approximate deliverable date
Perc
ent b
elow
targ
et
Mea
n sc
ore
All first year indicators over time
0 1 2 3 4 5 6-10 11-15
16-20
21-25
26-30
31-35
36-40
41-50
0
50
100
150
200
250
300
350
400
2642
67 65 6245
228
100
4125 10 2 2 0
344
187
7346 38
7 200 0 0 0 0 0 0
Below target Below threshold
Number of indicators
Num
ber o
f stu
dent
s# Students falling below expectations in first year
Graduating year• Starting point: histograms• Very few students falling below
threshold level in capstone courses for most indicators
46
3.02
GY1
3.02
GY2
3.02
GY3
3.02
GY4
3.04
GY1
3.04
GY2
3.04
GY3
3.04
GY4
3.04
GY5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Not demMarginalMeetsOutstanding
Area for improvement in graduating year: technical literature
FEAS - 3.12-FY1
FEAS - 3.12-FY2
FEAS - 3.12-FY5
FEAS - 3.12-FY6
0
10
20
30
40
50
60
1 - Not Demonstrated 2 - Marginal 3 - Meets ExpectationsAttributes
Per
cent
age
(%)
47
3.12-FY1 Uses information effectively, ethically, and legally to accomplish a specific purpose, including clear attribution of Information sources.
3.12-FY2 Identifies a specific learning need or knowledge gap.3.12-FY5 Identifies appropriate technical literature and other information sources to meet a need3.12-FY6 Critically evaluates the procured information for authority, currency, and objectivity.
Data evaluation
• Across multiple capstone courses, students scoring lower on indicators involving:• Evaluating validity of results• Evaluating techniques and tools• Evaluating effectiveness of results• Evaluating information
• Pattern: evaluation
48
Curriculum Mapping: CurriKit
• Curriculum mapping software developed by U Guelph
• Provides information to identify:• the courses which develop each
graduate attribute• what assessment is done and when• which instructional approaches are used
Program-wide assessment process flow
50Engineering Graduate Attribute Development (EGAD) Project
Defining Purpose and Outcomes
ProgramMapping
Stakeholder input
Collecting Data
Analysis andInterpretation
Create a ProgramImprovement Plan
Program & CourseImprovement
First year development • More focus on grader training and calibration• Problem analysis: greater focus on making an
effective argument• Design: the focus on safety and risk assessment• Communications: Weak communicators flagged
and supported in fall semester (EPT). Required resubmission/assistance for falling below threshold
• Lifelong learning: evaluating information• Ethics and equity and professionalism get stronger
emphasis
Development in upper years
• In 2012-2013 all programs will have an open-ended design experience in third year which will apply disciplinary tools and principles, and incorporate professional issues and communications
• In 2013-2014 capstone courses will be revised to strengthen professional skills (safety, role of engineers in protecting public safety, ethics, communications)
Program improvement• Students take more responsibility for
learning and demonstrating attributes• Faculty-wide curriculum development
• Engineering design and practice sequence
• Assessment built into sequence• Multi-disciplinary approach to
developing professional skills in an integrative experience
• Developing leadership through peer mentoring program
53
Process development• Improve common indicators• Develop disciplinary indicators• Triangulation: Indicators measured using
multiple methods or events to assess validity• Attributes measured at multiple times in
students' program with leveled expectations• Satisfy both CEAB and province (OCAV UDLEs
for Queen’s Quality assurance process)
54