Designing and Conducting Designing and Conducting Summative EvaluationsSummative Evaluations

Larry F. Cobb

ObjectivesObjectivesDefine the purpose of summative

evaluationDescribe the two phases of

summative evaluation and the decisions resulting from each phase

Design a summative evaluation to examine organizational benefits of instruction they have implemented

Contrast Formative and summative evaluation by purpose and design

What’s the Difference?What’s the Difference?

What is Summative EvaluationWhat is Summative Evaluation

Summative evaluation is defined as the design of evaluation studies and the collection of data to verify the effectiveness of instructional materials with target learners.

The Purpose of Summative The Purpose of Summative EvaluationEvaluation

Make “go-no-go” decisions• Keep current materials?• Look for something better suited to meet

organization’s specific instruction needs?• Summative evaluations are used to

judge the impact of a plan of instruction on the organization’s initial problem.

Evaluators – Why they should be Evaluators – Why they should be ExternalExternal

The primary evaluator in a summative evaluation is rarely the designer or developer of the instruction.

Evaluator is often unfamiliar with the materials, the organization requesting the evaluation, or the setting in which the materials are evaluated.

Preferred because they have no personal investment and will likely be more objective

Two Main Phases of Summative Two Main Phases of Summative EvaluationEvaluation

1.Expert Judgment Phase

2. Field Trial Phase

Expert Judgment PhaseExpert Judgment PhasePurpose:

Do the materials have the potential for meeting this organization’s needs?

There are several activities that decide whether the candidate instruction is promising.1. Congruence Analysis2. Content Analysis 3. Design Analysis4. Utility and Feasibility Analysis5. Current User Analysis

Expert Judgment PhaseExpert Judgment PhaseCongruence Analysis

Analyzing the congruence among1.An organization’s state needs and goals and those addressed in candidate instruction2.An organization’s target learners’ entry skills and characteristics and those for which candidate materials are intended3.An organization’s resources and those required for obtaining and implementing candidate instruction

Expert Judgment PhaseExpert Judgment PhaseContent Analysis

During this activity, an identified expert is used to judge material for accuracy and completeness to determine if they are inline with the organization’s stated goals. An instructional analysis of the stated goal is a very cost effective method.

Expert Judgment PhaseExpert Judgment PhaseDesign Analysis

The design analysis is an evaluation of the adequacy of the components of the instructional strategy included in the candidate material. Checklist are great during this activity.

Expert Judgment PhaseExpert Judgment PhaseUtility and Feasibility Analysis

Factors such as the availability of a learner guide or syllabus and an instructor’s manual are taking into consideration during this activity. This is also the time when you get information for the people that determine that the evaluation was necessary.

Expert Judgment PhaseExpert Judgment PhaseCurrent User Analysis

This final analysis seeks to get information about the candidate material from the organizations that are experienced in using them. The names of current users can often be obtained from the publishers of the materials.

Steps for Conducting Expert Steps for Conducting Expert Judgment Phase of Summative Judgment Phase of Summative EvaluationEvaluation

Field Trial PhaseField Trial PhasePurpose:

Are the materials effective with target learners in the prescribed setting?

Outcomes Analysis:1.Impact on Learners2.Impact on Job3.Impact on Organization

Field Trial Phase (continued)Field Trial Phase (continued)Management Analysis:1.Are instructor and manager attitudes

satisfactory?2.Are recommended implementation

procedures feasible?3.Are costs related to time, personnel,

equipment, and resources reasonable?

Steps for Conducting Field Trial Steps for Conducting Field Trial Phase of Summative EvaluationPhase of Summative Evaluation

Comparison of Formative and Comparison of Formative and Summative Evaluation (Table 12.3)Summative Evaluation (Table 12.3)

Formative Evaluation Summative Evaluation

PurposeLocate weaknesses in instructor in order to revise it

Document strengths and weaknesses in instruction in order to decide whether to maintain or adopt it

Phases or StagesOn-to-oneSmall groupField Trial

Expert judgmentField trial

Instructional Development History

Systematically designed in-house and tailored to the needs of the organization

Produced in-house or elsewhere not necessarily following a systems approach

Materials One set of materialsOne set of materials or several competing sets

Position of EvaluatorMember of design and development team

Typically an external evaluator

OutcomesA prescription for revising instruction

A report documenting the design, procedures, results, recommendations, and rationale

Change AgentChange AgentAs a change agent, is very important for

us to constantly evaluate the instructional strategy ensure that is staying the course with what the organization has in mind while being usable and feasible for the organization. The summative evaluation does just that.

Larry F. CobbLarry F. Cobblfcobb@alasu.edu