Evaluation Research and Engineering Education

Lesley Jolly

For AaeE

ljolly@bigpond.net.au

ERM wiki at www.aaee-scholar.pbwiki.com

What is evaluation research?

• Periodic assessment of results– Appropriateness, effectiveness, efficiency,

impact, sustainability

• Identifies intended and unintended effects

• Identifies what worked and what didn’t

• Provides level of judgement about overall worth of intervention

What has it got to teach us?

• Systematic approach to what we’re doing already

• Extracting lessons learned

• Legacy of sustained evaluation frameworks for ongoing data collection

• Arguments against ‘popularity contest’ course and program evaluation

Step 1: describe the program

• Aims and objectives– Need to be clearly articulated– Does everyone involved share the same aims

• Program logic diagram– Describes how we think the program

produces results– May be called logframe

Program logic identifies what we need to evaluate

Evaluating PBL tutor training

INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS

Training

materials

Staff time

Staff support

workshops

Rooms/ space

Library support

[no

Administrative

support]

Published

research

Compiling

materials

Putting

Materials

online

Running

workshop

Staff research

Review and

edit materials

Staff reflection

Numbers of

new tutors

Numbers of

Returning

tutors

Level of

satisfaction

Growing

library of

examples and

materials

Tutors facilitate

rather than

inform

Tutors

Understand

PBL

Tutors develop

Knowledge

independently

Tutors make

PBL -

Appropriate

contributions toCurriculum

review

Students take

Responsibility

for learning

Students learn

to focus

research and

Synthesise

results

Monitoring facilitates data collection

• Can be process (outputs) or impact (short to medium outcome) monitoring

• Need to develop indicators of progress• Targets may be included in indicators or

separate e.g.– In 2010 50% of 2nd yrs will have used the new facility

for more than 20 hrs– OR percentage of students using facility (indicator)– 50% in 2010 (target)

PBL Tutor training

• GOAL: provide timely well-placed supportive guidance to encourage tutors to scaffold and facilitate student learning

• OBJECTIVES: at the end of training the successful tutor will be able to

• articulate a good understanding of the objectives and methods of PBL

• guide student learning through providing appropriate support and guidance rather than information

• contribute to curriculum development within a staff team

Monitoring Tutor TrainingFOCUS PERFORMANCE

INDICATORSDATASOURCES

DATACOLLECTIONMETHODS

RESPONSIBILITY FOR COLLECTION

TIMEFRAME

Objective 1: tutors articulate a good understanding of the objectives and methods of PBL

OutcomesTutors facilitaterather thaninform

Tutor contributions to discussion

OnlineDiscussionlists

Content analysis of lists

Course co-ordinator Twice persemester

OutputsLevel ofsatisfaction

Likert: training hasclarified purposes of

PBL

Exit survey Paper survey post training

Trainer Everysemester

Objective 2: tutors guide student learning through providing appropriate support and guidance rather than information

OutcomesTutors facilitaterather thaninform

Students takeresponsibility forlearning

Tutor contributions to discussion

Student contributions to discussion

OnlineDiscussionlists

OnlineDiscussionlists

Content analysis of lists

Content analysis of lists

Course co-ordinator

Course co-ordinator

Twice persemester

Twice persemester

Evaluation asks formative and summative questions

• Are the questions cohesive and logical?

• Do evaluation questions link to monitoring data?

• Have ethical issues been addressed?

• What mechanisms are in place to gather the learnings generated by evaluation?

• What needs to be retained from this evaluation process in future years?

Vary questions to suit program

QUESTIONS SOURCES OF INFORMATION FROM MONITORING

SOURCES OFINFORMATIONFROM EVALUATIONDATA COLLECTION

DATA ANALYSISAND REPORTINGMETHODS

Is the program as implemented appropriate, and is it being appropriately monitored?

Has the purpose of the program been achieved?

Is the program being implemented in the most efficient and cost effective way possible?

Has the program produced sustainable results and outcomes?

PBL Tutor Training• Appropriateness

• Did the training model the target behaviour?

• Has the purpose of the training been achieved?

• Indicator 1: changes in Tutor behaviour indicating deeper knowledge of and commitment to PBL

• Indicator 2: changes in student behaviour

• Efficiency• Was the time invested by staff good value?

• Sustainability• what needs to retained as core material from year

to year to retain benefit of training

Owen, J. (2006) "Program Evaluation" Allen & Unwin: Crows Nest

Making use of evaluation

Dissemination and reporting

• Findings may be communicated throughout project to multiple audiences– Evidence, conclusions, judgements,

recommendations.

• Different occasions call for different styles– Oral, interactive workshops, posters, reports,

summaries, papers, conference presentations

• Must be well timed

Developing capacity

• “process use of evaluation”– Taking part develops skills– Taking part sensitises staff to issues

• Improved communication– Ongoing 360 degree dissemination– Development of local discourse