Joint Committee on Standards for Educational Evaluation

Evaluation Standards

Joan Kruger

Spring 2008

Joint Committee on Standards for Educational Evaluation

http:/www.wmich.edu/ evalctr/jc

SPONSORING ORGANIZATIONS:American Association of School AdministratorsAmerican Counseling AssociationAmerican Evaluation AssociationAmerican Educational Research Association

American Indian Higher Education Consortium American Psychological AssociationCanadian Evaluation Society

Canadian Society for the Study of Education

Consortium for Research on Educational Accountability and Teacher Evaluation

Council of Chief State School Officer

Council of the Great City Schools

National Association of Elementary School Principals

National Association of School Psychologists

National Association of Secondary School Principals

National Council on Measurement in Education

National Education Association

National Legislative Program Evaluation Society

National Rural Education Association

What is it?

Created in 1975, the Joint Committee (JC) is a coalition of major professional associations concerned with the quality of education

It is incorporated in the US as a private non-profit organization

It is accredited by the American National Standards Institute (ANSI) and all standards become certified by ANSI

What is its mission?

To promote concern for evaluations of high quality based on sound evaluation practices and procedures

To meet existing and emerging needs in the field of evaluation

Where is it located and what is the role of CES?

The Joint Committee is housed at the Evaluation Center, Western Michigan University

Canadian Evaluation Society has been a Sponsoring Organization since 1994 and participates as a voting member, participates in special projects, and promotes all Standards (Program, Personnel and Student) to CES members.

Direct Products of the JC

Publications: 3 sets of evaluation standards Personnel Evaluation Standards. 2nd ed. 2008.

Corwin Press Program Evaluation Standards. 2nd ed. 1994.

3rd in testing. Sage Student Evaluation Standards. 2003. Corwin

Press.

Associated Accomplishments

Adoption and adaptation by schools, states, and organizations nationally and internationally.

Numerous associated materials (e.g., Checklists for metaevaluation).

http://www.wmich.edu/evalctr/checklists/

Widespread translation and/or dissemination internationally, for use in Europe (especially Germany and Switzerland, Latin America, Africa and others)

Pervasive Nature of Program Evaluations

Primary tool for guiding program evaluation, crossing all academic disciplines and courses

Evaluations regularly clients and stakeholders in a wide variety of decision situations.

Impact of Program Evaluations

Evaluations done well can be of significant service.

Clients are victims of and harmed by poor evaluations in both high- and low-stake evaluations

Barriers to Sound Program Evaluations

Inadequate preparation Inadequate administrative and technical

support Professional disagreements Lack of shared language Inadequate policy

Why do we need Program Evaluation Standards?

What is available? Assessment &

measurement design Performance

management systems

Standards for student and personnel evaluation

What are the gaps? No standards for the

programs evaluations across disciplines.

No desktop reference to ensure quality

What is a Standard?

A principle commonly agreed upon by experts in the conduct and use of evaluation for the measurement of the value or quality of an evaluation. A standard is seen as a principle governing

good practice vs. a rule Technical guide to conduct evaluations Criteria of good practice for those who receive

and are affected by an evaluation

What is a Standard?

A guideline may provide the evaluator with a recommendation to help ensure basic quality evaluation. Other evaluators, in the same situation in the same or similar institutions following the guidelines, may also ensure basic quality.

The CES felt standards used as guidelines would help evaluators make more consistent quality judgements over time.

What is a Standard is not.

Rules are much more restrictive. The evaluator must or should conform to rules. This may lead to a narrower rating of a program or project by not allowing the evaluator to adapt the standard to the norms of the evaluation setting.

Rules are usually policed by a professional body, and offenders may be reprimanded.

How are the Standards Organized?

All three sets of Standards are organized into four main categories:

Utility Propriety Feasibility Accuracy

Utility

To ensure that an evaluation will serve the information needs of intended users.

Propriety

To ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation as well as those affected by its results.

Additional resource: Canadian Evaluation Society Ethical Guidelines

http://www.evaluationcanada.ca/site.cgi?s=5&ss=4&_lang=EN

Feasibility

To ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.

Accuracy

To ensure that an evaluation will reveal and convey technically adequate information bout the features that determine worth and merit of the program being evaluated.

How are the Standards Presented in the Book?

A uniform pattern is followed: Standard Statement Overview (Explanation, Rationale, Caveats) Guidelines Common Errors Two illustrative case studies Supporting documentation (references)

Number of Standards(Program: proposed 3rd ed)

Category Program Personnel Student

Utility 8 5 7

Propriety 7 5 7

Feasibility 4 3 3

Accuracy 8 8 12

TOTAL 27 21 29

National Public Hearings

National Public Hearings: The Joint Committee will hold open forums at meetings of each of the sponsoring organizations to gain additional feedback from members of these organization (See 5.3.5 of the Operating Procedures-).

5.3.5 National Public Hearings National Public Hearings shall be held to provide an open forum for discussion and

critique of the standards. Hearing schedules shall be publicized and all interested parties will be encouraged to participate. In addition, each Sponsoring Organization will be asked to select members of their organization and encourage them to respond to the draft through the vehicle of the hearings. (NOTE: If the standards under consideration are to be submitted to the American National Standards Institute for approval as American National Standards, the Joint Committee shall also arrange for announcement of the draft in ANSI's STANDARDS ACTION for comment, in accordance with Section 1.2.6 of the ANSI Procedures for the Development and Coordination of American National Standards.)

Process for Hearing

Overview of the Standards Comments from CES members and

others (criticisms, observations, recommendations)

Comments from Field test participants

Further Communication

Visit the JC web site at

http://www.wmich.edu/evalctr/jc/ and enter ES review page.

Contact Arlen Gullickson arlen.gullickson@wmich.edu

Next Steps

Complete field tests Complete National Hearings Revise Final Approval by the Joint Committee Submission to American National

Standards Institute for certification Publish

Evaluation Web Sites

Canadian Evaluation Society (CES)

http://www.evaluationcanada.ca/

Canadian Evaluation Society Educational Fund (CESEF) – for students and those new to eval.

http://www.evaluation-education.org/index.html

Saskatchewan Chapter, CES

http://sk.evaluationcanada.ca/

American Evaluation Association (AEA)

http://www.eval.org/