PROGRAM EVALUATION

GNRS 5311 MODULE 4

Meredith French, BSNDebra Morris, BSNAnne Nance, BSNChloé Williams, BSN

OBJECTIVES

Define program evaluation in nursing education.

Discuss theory-driven and method-driven evaluation methods.

Understand how the complexity theory drives a program evaluation.

Describe the elements necessary to evaluate each component of the evaluation plan.

Analyze the specific criteria determined by the Commission on Collegiate Nursing Education (CCNE).

PROGRAM EVALUATION•Required for accreditation and approval by State Board of Nursing (Pross, 2010)

•Driven historically by the accreditation process (Sauter et al., 2012)

•Link to and integration of accreditation programs are still in use (Kalb, 2009; Escallier & Fullerton, 2012)

•Key relationships of elements can be overlooked

PROGRAM EVALUATION THEORIES

•Underlying theoretical foundation

•Method-oriented versus theory-driven

•Method-oriented approaches have been popular (Sauter et al., 2012)

•Method-oriented ignore views and concerns of stakeholder’s (Chen, 2012)

•Theory-driven models should be considered

THEORY-DRIVEN PROGRAM EVALUATION

•Tests correct program theory and successful implementation

•More scientifically sound, with causal explanations and involved mechanisms (Coryn et al., 2011)

•Information generated improves existing or future programs (Chen, 2012)

•Chen’s theory-driven model used for nursing programs (Sauter et al, 2012)

THEORY-DRIVEN PROGRAM EVALUATIONChen’s Evaluation

Normative outcome evaluationDevelop program theoryDefine activities to achieve goalsExplain linkages between programsIdentify/prioritize goals & outcomes

Normative treatment evaluationEvaluate congruency between expected and implemented treatment

Evaluation-Nursing Education Programs

Mission & goal evaluationDevelop mission, philosophy & framework

Identify program goals & outcomesEnsure program goals & outcomes meet professional standards

Program evaluation

Evaluation of teaching effectiveness

Adapted from Sauter, M.K., Gillespie, N.N., & Knepp, A. (2012). Educational program evaluation. In D.M. Billings & J.A. Halstead (Eds.), Teaching in nursing (p 507-508).

THEORY-DRIVEN PROGRAM EVALUATIONChen’s Evaluation

Implementation Environment evaluationParticipant evaluationImplementer dimensionDelivery mode dimensionImplementing organization dimension

Evaluation-Nursing Education Programs

Environment EvaluationStudent dimensionFaculty dimensionDelivery mode dimension including courses

Organization dimension including leadership

Adapted from Sauter, M.K., Gillespie, N.N., & Knepp, A. (2012). Educational program evaluation. In D.M. Billings & J.A. Halstead (Eds.), Teaching in nursing (p 507-508).

THEORY-DRIVEN PROGRAM EVALUATION

Chen’s Evaluation

Implementation Environment evaluationInter-organizational relationshipsImpact evaluationIntervening mechanismsGeneralization evaluation

Evaluation-Nursing Education Programs

Environment EvaluationInter-organizational relationshipOutcomes assessmentIntervening mechanismsGeneralization evaluation

Adapted from Sauter, M.K., Gillespie, N.N., & Knepp, A. (2012). Educational program evaluation. In D.M. Billings & J.A. Halstead (Eds.), Teaching in nursing (p 507-508).

COMPLEXITY THEORY

• Theory-driven evaluation method

• Less rigid approach• Looks at the program in it’s

entirety(Frye & Hemmer, 2012)

COMPLEXITY THEORY

•Ever-changing knowledge and practice patterns, therefore nursing programs are complex systems

•A program as a whole is greater than the sums of its parts

•Allows educators to …• structure useful program evaluation that accommodate a program’s true complexity• avoid an overly narrow approach

•Provides a useful perspective for choosing an evaluation theory that serves the program more effectively (Frye & Hemmer, 2012)

COMMISSION ON COLLEGIATE NURSING EDUCATION (CCNE)

• Specific criteria for evaluation:

Baccalaureate

Master’s

Doctor of Nursing Practice (DNP)

Post-graduate Advanced Practice Nurse

•Self-study

•Standards for accreditation

EVALUATION COMPONENTS

•Purpose of program evaluation

Mission

Goals

Outcomes

Implementation

Commitment

Resources

Curriculum/Teaching-Learning

UNIVERSITY OF TEXAS MEDICAL BRANCH

CCNE

•Required documentation

Standards and guidelines

Official documentation

Appointment, promotion, and tenure

Unit reports

Official correspondence

Handbooks, manuals, calendars, etc.

Advertising

CCNE: RESOURCES

•Budget/Resources

Nursing personnel

Location space

Equipment/Supplies

•Degree Requirements

Full-time faculty

Student-ratio

•Supporting documentation

Budget

Degree/certification/experience

CV

Bargaining/policies/documents

CCNE: TEACHING-LEARNING

•Curriculum

•Objectives

•Outcomes

•Nursing standards/guidelines

•Teaching-learning

•Environment

•Competencies

•Required documentation

Syllabi

Didactic/clinical learning

Student performance

Evaluations

Affiliation agreements

Documents

Assignments

Formal data and licensure

CCNE: ASSESSMENT

•Assessment and achievement

•Quantitative/qualitative

•Completion rates

•Licensure/certification rates

•Employment rates

•Formal complaints

•Required documentation

Completion rate (each degree)

Faculty outcomes

Formal complaints

Documents

Data

CONCLUSION

REFERENCES•Chen, H. T. (2012). Theory-driven evaluation: conceptual framework, application and advancement. In R. Strobl, O. Lobermeier, & W. Heitmeyer (Eds.), Evaluation von programmen und projekten fur eine demokratische kultur (pp. 17-40). Weisbaden: Springer. doi:10.1007/978-3-531-19009-9_2

•Commission on Collegiate Nursing Education (2013). Retrieved from http://www.aacn.nche.edu/ccne-accreditation/Standards-Amended-2013.pdf

•Coryn, C.L.S., Noakes, L.A., Westine, C.D., & Schroter, D.C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199-226. doi: 10.1177/1098214010389321

•Doll Jr, W.E. & Trueit, D. (2010). Complexity and the health care professions. J Eval Clin Pract. 16(4), 841-848.

•Escalier, L.A., & Fullerton, J.T. (2012). An innovation in design of a school of nursing evaluation protocol. Nurse Educator, 37(5), 187-191. doi: 10.1097/NNE.0b013e318262eb15

•Frye, A.W., & Hemmer, P.A. (2012) Program evaluation models and related theories: AMEE guide no. 67. Medical Teacher, 34, 288-299. Doi:10.3109/0142159X.2012.668637

•Kalb, K.A. (2009). The three Cs model: the context, content, and conduct of nursing education. Nursing Education Perspectives, 3(30), 176-180.

•Sauter, M.K., Gillespie, N.N., & Knepp, A. (2012). Educational program evaluation. In D.M. Billings & J.A. Halstead (Eds.), Teaching in nursing (pp. 503-549).