State Evaluation Policies in Gifted Education: Examining Options for Policymakers

8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers

1/27

State Evaluation Policies in Gifted Education:

Examining Options for Policymakers

Kristina Ayers Paul

B.S., Eastern University, 2000

M.A., Rosemont College, 2003

A Dissertation Proposal

Submitted in Partial Fulfillment of the

Requirements for the Degree of

Doctor of Philosophy

at the

University of Connecticut

2009


2/27

Abstract

Program evaluation in gifted education is a highly recommended yet underutilized

practice for ensuring high quality, effective services for gifted learners. State policy plays an

influential role in the programmatic decisions and practices of gifted programs within each state,

including evaluation. It often serves as a bridge from recommendation to practice. This policy

analysis study will explore the extent to which states have policies that promote gifted program

evaluation and examine the different products that emerge from the interpretation of these

policies (e. g. monitoring systems, self-assessment tools, resource manuals). Policy documents

will be collected and analyzed to determine the prevalence and types of evaluation policies

currently in use among the 50 states, while purposefully selected case studies will explore the

features and quality of each type policy. The goal of this research is to enhance our

understanding of state evaluation policy in gifted education by offering a sweeping view of the

national landscape and a focused look at each type of existing state policy, providing an

overview of the options available to policymakers, decision-makers, and advocates of gifted

program evaluation.


3/27

State Evaluation Policies in Gifted Education:

Examining Options for Policymakers

One of the conditions of a good policy analysis is that it is helpful to a decision maker.

A decision maker looks at it and finds he or she understands the problem better, understands the

choices better, or understands the implications of choice better. (Lynn, 1980, p. 85).

While program evaluation is highly recommended for ensuring high-quality gifted

programs and state policies provide a bridge from recommendation and practice, there is mystery

surrounding the use of evaluation in gifted programs. A number of state gifted education policies

include provisions that influence the evaluation of gifted programs, so we know that evaluation

is occurring at some level. What we do not know is the types of evaluations that programs are

asked to conduct, the usefulness of these evaluations, or the quality of these evaluations. With

the pivotal leveraging role that state policy plays in the programmatic decisions and practices of

gifted programs at the local level, it is pertinent to understand the nature of these policies. By

examining the evaluation practices promoted through state policy, we can begin to understand

the context within which local gifted programs are asked to engage in evaluation activities.

This policy analysis will build a taxonomy to describe the range of state evaluation policy

models currently in use. The features of each policy model will be demonstrated through case

studies that provide a focused look at the utility of the policy products that emerge from

interpretation at the state level. In addition, each model will be evaluated against a matrix of

indicators of quality gifted education policy and gifted program evaluation. The results of this

study will provide advocates, decision-makers, and policymakers with options for using state

policy to leverage the practice of gifted program evaluation at the local level.

Statement of the Problem

From the boardroom to the classroom, we are witnessing a social epidemic of

evaluation (Preskill, 2008, p. 129). Decision-makers across every sector have renewed their


4/27

focus on questions such as:How do we know if this program works?How can we ensure that the

time, money, and resources devoted to this program are well spent? How will we determine if

this program accomplishes what it is meant to do?

The best practice literature from the past three decades in gifted education

demonstrates a solid appeal for the use of program evaluation to answer these questions. Since

the publication of the first manual for gifted program evaluation (Renzulli, 1975), a number of

other manuals, service publications, book chapters, and articles have been published to

recommend how gifted and talented programs can use evaluation as a tool for making data-

driven decisions about program changes or improvements (e.g. Callahan, 2009; Callahan &

Caldwell, 1997; Fetterman, 1993; Park, 1984; VanTassel-Baska & Feng, 2004). In fact, program

evaluation standards have even been included in the National Association for Gifted Childrens

(NAGC) PreK-12 Gifted Program Standards (Landrum & Shakelee, 1998), signaling the value

that the field places on the use of program evaluation as an indicator of a high-quality gifted

education program.

Despite these recommendations for incorporating program evaluation as an important

practice in gifted programs, the actual frequency with which gifted programs engage in

evaluation is not well understood (Avery & VanTassel-Baska, 2001). Three meta-evaluations of

evaluation reports have provided some understanding of the nature of evaluation practice in

gifted programs (e.g. Callahan, Tomlinson, Hunsaker, Bland, & Moon, 1995; VanTassel-Baska,

2004, 2006a) and a handful of published articles describe case studies of gifted program

evaluations (e.g. Avery & VanTassel-Baska, 2001; Avery, VanTassel-Baska, & ONeill, 1997;

Hertzog & Fowler, 1999), yet the sum of this work provides a murky picture of the status of

gifted program evaluation across our nation. Some leaders in the field have concluded that

quality program evaluation rarely occurs (VanTassel-Baska, 2006b; Callahan et al., 1999),

although the paucity of research and published examples of gifted program evaluation limits the

degree to which such a criticism can be made.


5/27

A number of state education agencies (SEAs) have policies that influence the use of

evaluation in the gifted programs within their borders (NAGC & Council of State Directors of

Programs for the Gifted [CSDPG], 2009). Brief descriptions of some of these policies are

included in Table 1. Based on what we know states are requiring local gifted programs to do, we

can conclude that program evaluation is happening at some level; nevertheless, we do not have a

comprehensive picture of what these policy-inspired evaluations entail.

Several researchers have briefly examined policies that influence program evaluation

practices within the context of larger studies of comprehensive state policies in gifted education

(e.g. Brown, Avery, VanTassel-Baska, Worley, & Stambaugh, 2006; Clinkenbeard, Kollof, and

Lord, 2007;). Gallagher (2002), a leading voice in advocacy for gifted education policy, suggests

that program evaluation is one of the four most needed new policies in gifted education.

However, there is no known study that fully explores the nature of existing state evaluation

policies in gifted education.

The purpose of this policy analysis study is to explore the various state policy models that

influence program evaluation practices at the local level. Through the lens of utility, it is my goal

to reveal the features of the various policy models currently used, explore the products that result

through the interpretation of the policies at the state level (e. g., guidelines, monitoring systems,

self-assessment forms), and evaluate the quality of each policy type against a matrix of criteria

gleaned from the best practices literature in gifted education policy and gifted program

evaluation. This study marks an important first step in understanding the policy context within

which gifted program evaluations occur.

Table 1:

Brief Descriptions of State Policies Influencing the Practice of Evaluation in Gifted Programs

State Data Sources Brief Description of Policy

Arizona State of the States 2009 Arizona House Bill 2552 (2006)

LEAs must conduct annual program evaluations andsubmit the results to the SEA; On-site monitoring isconducted as an integrated component of the Title Imonitoring schedule.


6/27

Hawaii Program Guide for Gifted andTalented

School Gifted and Talented ProgramChecklist

Chapter 51 Regulations

Programs are to conduct annual evaluations for programimprovement, which are to be approved by the ComplexSuperintendent.

Maine Guidelines for the Gifted and TalentedApplication, Budget, Academic Plan,and Visual and Performing Arts Plan

Application for Gifted and TalentedEducational Program Approval andApproval of Program Costs for State

Subsidy Visual and Performing Arts Plan for

Gifted and Talented 2009-2010 Academic Plan for Gifted and

Talented 2009-2010

LEAs must annually submit a program plan/applicationthat must be approved by the SEA. The plan/applicationmust include a summary of the results of programevaluation and a plan for how the program will beevaluated in the upcoming year.

Mississippi Regulations for the Gifted EducationPrograms in Mississippi (2006)

Annual Self-Evaluation of the LocalGifted Education Program: Directionsand Sample Instrument (2007)

Mississippi Gifted EducationPrograms Program StandardsEvaluation & Monitoring (2004)

LEAs must annually submit the results of a programevaluation using the state-supplied self-assessment; Mustsubmit a plan for corrective action on the items that do notmeet the minimum standard set by the state.

Pennsylvania 22 Pa. Code Chapter 16 Basic Education Circular: Monitoring

Schedule for Gifted Education (2009) Gifted Education Program Self-

Assessment Instrument (2003)

At least ten on-site compliance-monitoring visits per yearare conducted annually using a facilitated self-assessment,resulting in improvement and/or corrective action plans.

Texas Texas State Plan for the Education ofGifted/Talented Students (2000)

Flyer: Level 3 School BoardTraining

Advanced Programs & Gifted andTalented Education Questions andAnswers on the State Plan

Local school boards are required to monitor programcompliance and are provided with a standards-based rubricand training opportunities for evaluating compliance of the

program.

Background

Definition of Policy

Definitions of policy abound, all intended to describe the money, rules, and authority

that are used to influence the actions of individuals and institutions (McDonnell & Elmore,

1987, p. 133). According to Gallagher (2002), social policy is defined by the allocation of

resources (e.g., money, human) to any given cause within the broad scope of social needs.


7/27

Gallagher suggests that evidence for social policy should be derived from four sources:

legislation, courts, administrative rules, and professional standards.

However, within the context of the current study, I am aiming to describe the ways that

local gifted programs are influenced by the rules, regulations, and guidelines of their state, for as

VanTassel-Baska (2006b) suggested, The structure that holds gifted programs together is nested

in the policies that individual states have enacted. The direction and continuity of the field at the

grassroots level, then, are heavily influenced by the state one resides in and the strength of the

policy initiatives in the state (p. 250). Therefore, a more targeted definition of policy within the

context of this study is the rules, statutes, codes, and regulations adopted by state legislatures,

interpreted by state school boards of education and state departments of education, and

implemented by local school districts (Brown et al., 2006, p. 11).

Policy in Gifted Education

The body of knowledge regarding state policy in gifted education is not an unexplored

territory-maybe untended, but not unexplored (Clinkenbeard et al., 2007, p. 2). However, the

focus of state policy inquiry has centered primarily on: (a) comprehensive state gifted education

policy (Brown et al., 2006; Clinkenbeard et al.; NAGC & CSDPG, 2009; Passow & Rudnitski,

1993;Swanson, 2002); (b) focused examinations of state policies regarding the identification ofgifted students (Coleman & Gallagher, 1995); (c) focused examinations of state policies

regarding acceleration in schools (Institute for Research and Policy on Acceleration [IRPA],

NAGC, and CSDPG, 2009); and (d) the allocation of state funding for gifted education (Baker,

2001; Baker & McIntire, 2003).

In 2007 NAGC published a seminal document for gifted education policy, which

unveiled the efforts of the NAGC policy task force to collect sample policy language

(Clinkenbeard et al., 2007). This document provides a discussion of the four major types of

gifted education policy: (a) identification, (b) program and curriculum, (c) personnel preparation,


8/27

and (d) program management: assessment/evaluation. Within the discussion of each type of

policy, different models are explored and examples of exact state policy language are shared.

Additionally, hyperlinks to the policies are included. Policies related to program evaluation were

categorized under the umbrella of Program Management: Accountability/Evaluation. There are

four types of policies under this umbrella, which are outlined in Table 2.

Table 2:

Four Types of Policy forProgram Management: Accountability/Evaluation

(Clinkenbeard et al., 2007)

1. Review of local education agency (LEA) plans for gifted education by the state

education agency (SEA);

2. Requirements for LEA plans to include some or all of the following components:

a. screening, identification, and referral processes

b. program provisions employed at each grade level, K-12c. goals and student outcome assessment process for each program model

d. contact time for each model

e. pupil-teacher ratios for classes

f. gifted teacher planning time

g. professional development plans for educators working with gifted learners

h. counseling and guidance

i. program evaluation design

3. SEA monitoring plans that includes all LEAs and require on-site visits; and

4. Annual reports from the LEAs to the SEA to report progress and include student and

program evaluation data.

The NAGC Policy Guide touches on the subject of evaluation policies within the context

of the larger policy umbrella; however, a deeper analysis of this type of policy is warranted. As

previously stated, there are no known studies that specifically focus on state evaluation policy in

gifted education. Evaluation policy is defined as policies such as (but not only) high-level rules

embedded in legislation that are used to guide the practice of evaluation (Mark, Cooksy, &

Trochim, 2009, pp. 4-5). The lack of attention to evaluation policy is not surprising, given the

general paucity of literature on this topic (Mark et al.). Accordingly, a recent issue ofNew


9/27

Directions in Evaluation (Trochim, Mark, & Cooksy, 2009), a peer-reviewed, themed journal of

the American Evaluation Association, was dedicated to calling researchers attention to the void

of empirical knowledge on evaluation policy, recommending an agenda for furthering

knowledge, and highlighting several tools and models to aid researchers in developing this area

of research. One of these tools has been incorporated into the conceptual framework for this

study, which is discussed later in this proposal.

Policy Analysis

There are generally three choices for the evaluand in a policy analysis: process, product,

and performance (Gilbert & Terrell, 2005). When analysts examine policyprocess, they seek to

understand how and why the policy developed. Questions such as, What factors lead to the

development of such policy? and, What were the dynamics of the context and politics

surrounding the development of the policy? guide the study of policy process. When analysts

examine theproductof policy, they seek to understand the resulting language, systems,

documents, support structures, etc. that result from the policy itself and the interpretation of the

policy. Questions such as, What is the content of the policy statement? and What are the

resulting structures or systems that are created as a result of the policy?guide the study of

policy product. Finally, when analysts examine theperformance of policy, they seek to

understand how the influence that the policy has had in implementation. Questions such as, Is

the policy accomplishing what was intended? and What effect does the policy have on relevant

stakeholders?guide the study of policy performance.

Trochim (2009) outlined several key approaches to the analysis of evaluation policy.

Within this newly developing area of study, he indicated that the first priority should be for

developing a basic understanding of the types of evaluation policies that are currently in use.

We desperately need descriptive studies of current evaluation policies, and this would be


10/27

something we could get to work on immediately (p. 28). He recommended that systematic

reviews of policy documents be conducted to identify existing evaluation policies. In other

words, he recommended the evaluation policyproducts as the starting point, suggesting that

these products could be categorized within a taxonomy of evaluation policy. The taxonomy

could be used for, auditing the as-is situation and suggesting potential gaps (p. 28).

Two recent policy studies provide useful models for the study of state policy products in

the manner that Trochim (2009) suggested. Bocala, Mello, Reedy, and Lacireno-Paquet (2009)

and Stepanek and Peixotto (2009) studied state policies regarding response to intervention (RTI)

in schools. These studies elicited descriptions and features of the policy models that are used by

states in the Northwest and Northeast United States. While Bocala et al. used publicly available

documents to identify the features of the RTI policy models used by the Northeastern states,

Stepanek and Peixotto also incorporated interview data from key state education agency

personnel who managed the states RTI policies and initiatives. Each of these studies explored

the different models and compared them to core elements of RTI as described by the literature.

However, these researchers were careful to limit their analysis and conclusions to the face value

of the policy models, recommending that future research concentrate on the contextual factors

that influence these policies (policy process) and the implementation of these policies at the local

level (policy performance). Nevertheless, these studies mark an important first step in

understanding how SEAs are influencing the practice of RTI in schools.

Research Questions

In the spirit of Trochims (2009) recommendation for descriptive studies of existing

evaluation policies, this study will approach the analysis of state evaluation policy in gifted

education in terms of the product: that is, the content of the policy and the products that emerge

from the interpretation of these policies. Once this baseline of information has been established,

additional studies will be pursued to explore the implementation and effectiveness of the


11/27

different types of policies (policy performance), as well as the contextual factors that influence

the policy process.

This study will be guided by three questions:

1. What are the existing models of state evaluation policy in gifted education?

2. What are the key features of each model of state evaluation policy in gifted

education?

3. What resources and activities are in place at the state level to support the various

models of evaluation policy in gifted education?

4. How does each state evaluation policy model in gifted evaluation align with

indicators of utility and quality?


12/27

Conceptual Framework

A conceptual framework has been developed to guide this study and is depicted in Figure

1. This conceptual framework is rooted in several important elements, which provide lenses for

examining the features of each type of policy examined.

Figure 1: Conceptual Framework for the Analysis of State Evaluation Policies in Gifted

Education

The Evaluation Policy Wheel

Another of Trochims (2009) key recommendations for the study of evaluation policy is

to categorize the different types of policies using a taxonomy. He proposed a preliminary

taxonomy, called the Evaluation Policy Wheel (EPW), although he recommended that the

taxonomy could be adapted and modified as the body of knowledge grows.

The EPW is a multi-layered wheel that provides a system for categorizing and analyzing

different types, levels, and elements of evaluation policy. The EPW has eight pie slices that

describe a different type or focus of evaluation policy, including policies for: (a) evaluation

goals, (b) participation in evaluation, (c) evaluation capacity building, (d) evaluation

management, (e) roles in evaluation, (f) evaluation process and methods, (g) evaluation use, and

(h) meta-evaluation.

Another feature of the Evaluation Policy Wheel (EPW) is the ringed nature of the wheel,

similar to that of a bulls eye target. This serves to illustrate the range of policy generality or

specificity, with broad policy statements on the outside of the wheel and the more specific,

practical application of those policies in the center. As broad policies move through the


13/27

(sometimes) many layers of interpretation, trickling down to application at the local level, they

can be placed on the wheel in the appropriate ring of specificity within each pie-slice category.

The first layer of analysis in this study will result in a taxonomy that will characterize the

as-is situation of state evaluation policy in gifted education. Using the EPW as a starting point,

the taxonomy will be modified to address the specific context of policy in gifted education. The

pie slices, or categories, will reflect the different types of policies currently in use by states,

while the rings of the EPW will be used to pinpoint the movement of policy products from the

more general to the more specific.

Utilization-Focused Evaluation

As the field of program evaluation has evolved, the strategies, models, and approaches to

evaluation have become more sophisticated (Stufflebeam, 2001). Among the 22 most popular

approaches examined by Stufflebeam in his seminal meta-evaluation of evaluation approaches,

several emerged as particularly worthy of continued practice. One of these, Utilization-Focused

Evaluation (Patton, 2008), seems particularly appropriate to use as a lens through which the

usefulness of policy products can be examined.

The main aim of Utilization-Focused Evaluation (U-FE) is to keep the intended uses and

users of the evaluation central in all methodological decisions. The hallmarks of useful

evaluations, according to Patton (2008), are as follows:

Clear identification of the primary intended uses of the evaluation,

Clear identification of the primary intended users of the evaluation,

Clear identification of the evaluation criteria,

Establishment of a clear action framework,

Distinction between the empirical questions and the values questions,

Selection of methods that are appropriate for the questions, and

Facilitation of use of findings.


14/27

The second layer of analysis in this study will examine each type of evaluation policy in

terms of its utility. Each type of policy and resulting policy products (e.g. documents, resources,

activities) will be examined through the lens of U-FE, using the aforementioned hallmarks of

useful evaluation.

High-Quality State Evaluation Policy in Gifted Education

The third layer of analysis will focus on elements of quality. There are two tools that will

be used to examine quality: VanTassel-Baskas Attributes of High-Quality Gifted Education

Policy (2006) and the NAGC PreK-12 Gifted Program Standards for Program Evaluation

(Landrum & Shakelee, 1998). Each policy type will be examined for these indicators of quality,

and a cross-case matrix analysis will be completed for cross comparison. Appendix A includes

the matrix that will be used for this analysis.

Methods

This policy analysis study will begin with a review of the State of the States in Gifted

Education 2008-2009 report (NAGC & CSDGP, 2009) and the NAGC Policy Guide

(Clinkenbeard et al., 2007). Those states that have no policies or mandates for gifted education

will immediately be excluded from the sample. The data from the remaining states will be

examined to determine which of them have some form of policy influencing evaluation of gifted

programs. These states will form the basis for the initial sample from which I will conduct a

search of all publicly available information and documents that represent gifted education

policies.

The next step will be a content analysis (Krippendorff, 1980) using deductive coding

(Patton, 2002). The codebook will be developed from the EPW. This analysis will result in a

taxonomy of state evaluation policy in gifted education similar to the EPW. This taxonomy will

provide a clear picture of the different models of evaluation policies currently in use by SEAs.

Once I have determined the models of evaluation policies in use, I will purposeful

sampling (Creswell, 2007) to select specific state policies as case studies (Yin, 2009) that will


15/27

illustrate each type of policy. I will conduct document analyses, surveys, and semi-structured

follow-up interviews to probe each case; however, the flexibility of an emergent design will be

retained to allow for additional methods of data collection to be used as the need for them

emerges (Patton, 2002). Sample questions that may be used in the survey and follow-up

interview are included in Appendix B and Appendix C.

As recommended by Miles and Huberman (1994), I will use a pre-structured case outline.

A pre-structured case outline is appropriate in case study research when research questions have

been well specified. It prevents the researcher from becoming overwhelmed by the amount of

data available in case study data collection. This pre-structured case outline will provide the

framework needed within the case narrative to describe the features of each policy model,

incorporating elements from level two of the conceptual framework. Rich, thick description

(Merriam, 2009) will be used to complete each case outline, which will then be shared with the

informant to ensure that the policy and policy products have been accurately described (i.e.,

member check).

Upon receipt and incorporation of feedback (as warranted) from the informant, I will

complete a matrix analysis of each case to examine the quality of the policy as described in the

conceptual framework. The draft matrix is provided in Appendix A. As an added layer of

trustworthiness, a critical friend (Merriam, 2009) will read the case descriptions and complete

the matrix analysis. Following each of our analyses, we will debrief and reconcile disparities

between the ratings through a consensus-building discussion (Miles & Huberman, 1994).

Cross-case data displays will be constructed, not to compare or rank the different models,

but rather to provide a visual display of the various features of each policy model. The intention

for these displays is to provide a clear picture of the options available to policymakers. Examples

of these data displays are included in Appendix A and Appendix D.

This study should result in improved understanding of the various state policy models

that influence the practice of evaluation in gifted programs. The intent is that this study will help


16/27

policy makers understand the problem better, understand the choices better, or understand the

implications of choice better (Lynn, 1980, p. 85).

Limitations and Future Directions

While the purpose of this policy analysis is to examine the policy models themselves,

additional research is needed to explore the effectiveness of each policy model. It would also be

important to conduct in-depth explorations of the context within which each of these models is

best suited, as research has suggested that the political culture of each state may play a mediating

role in the policy and leadership practices at the state, district, and local level (Louis, Thomas,

Gordon, and Febey, 2008).

Another important future direction within this line of research is a focus on the economic

factors that influence states gifted education evaluation policies. The recently published State of

the States report (NAGC & CSDGP, 2009) highlighted the tenuousness of funding in gifted

education policy. An important direction of this research might be to explore the ways in which

state-initiated gifted program evaluations might actually provide the evidence needed to

command the attention of decision-makers in charge of appropriating funds for educational

programs. Some theoretical literature has explored the signaling role that program evaluation

plays in alerting government to worthwhile programs and subsequent resource allocation (Henry

& Mark, 2003; Segerholm, 2003; Trochim, 2009). Attention to evaluation as a tool for

advocating for mandates and funding at the state level might provide valuable information to our

field.


17/27

References

Avery, L. D., & VanTassel-Baska, J., (2001). Investigating the impact of gifted education

evaluation at state and local levels: Problems with traction.Journal for the Education of

the Gifted, 25, 153-176.

Avery, L. D., Vantassel-Baska, J., & ONeill, B., (1997). Making evaluation work: One School

Districts Experience. Gifted Child Quarterly, 41(4), 124-131.

Bocala, C., Mello, D., Reedy, K., and Lacireno-Paquet, N. (2009). Features of state response to

intervention initiatives in Northeast and Islands Region states (Issues & Answers Report,

REL 2009No. 083). Washington, DC: U.S. Department of Education, Institute of

Education Sciences, National Center for Education Evaluation and Regional Assistance,

Regional Educational Laboratory Northeast and Islands. Retrieved from

http://ies.ed.gov/ncee/edlabs .

Brown, E., Avery, L., VanTassel-Baska, J., Worley, B. B., & Stambaugh, T., (2006). A five-state

analysis of gifted education policies.Roeper Review, 29, 11-23.

Callahan, C. M. (Ed.). (2004). Program evaluation in gifted education (Vol. 11). In S. M. Reis

(Series Ed.),Essential readings in gifted education. Thousand Oaks, CA: Corwin Press.

Callahan, C. M. (2006). Developing a plan for evaluating a program in gifted education. In J. H.

Purcell & R. D. Eckert (Eds.),Designing services and programs for high-ability

learners: A guidebook for gifted education (pp.195206). Thousand Oaks, CA: Corwin

Press.

Callahan, C. M. (2009). Evaluating for decision-making: The practitioners guide to program

evaluation. In J. S. Renzulli, E. J. Gubbins, K. S. McMillen, R. D. Eckert, & C. A. Little

(Eds.), Systems and models for developing programs for the gifted and talented(2nd ed.).

Mansfield Center, CT: Creative Learning Press.
http://ies.ed.gov/ncee/edlabshttp://ies.ed.gov/ncee/edlabs


18/27

Callahan, C. M., & Caldwell, M. S. (1997).A practitioners guide to evaluating programs for

the gifted. Washington, DC: NAGC.

Callahan, C. M., Tomlinson, C. A., Hunsaker, S. L., Bland, L. C., & Moon, T. (1995).

Instruments and evaluation designs used in gifted programs (Research Report No.

95132). Retrieved from the National Research Center on the Gifted and Talented website:

http://www.gifted.uconn.edu/nrcgt/nrconlin.html#95132

Clinkenbeard, P. R., Kolloff, P. B., & Lord, W. E. (2007).A guide to state policies in gifted

education. Washington, DC: National Association for Gifted Children.

Fetterman, D. M. (1993).Evaluate yourself. (Report No. 9304). Retrieved from the National

Research Center on the Gifted and Talented website:


Gallagher, J. J. (2002). Societys role in educating gifted students: The role of public policy.

(Report No. RM02162). Retrieved from the National Research Center on the Gifted and

Talented website: http://www.gifted.uconn.edu/nrcgt/nrconlin.html#02162

Gilbert, N., & Terrell, P. (2005).Dimensions of social welfare policy (6th ed.). Boston, MA:

Allyn & Bacon.

Henry, G. T., & Mark, M. M., (2003). Beyond use: Understanding evaluations influence on

attitudes and actions.American Journal of Evaluation, 24, 293-314.

Hertzog, N. B., & Fowler, S. A., (1999). Perspectives: Evaluating an early childhood gifted

education program.Roeper Review, 21, 222-227.

Institute for Research and Policy on Acceleration, National Association for Gifted Children, &

Council of State Directors of Programs for the Gifted, (2009). Guidelines for developing

an academic acceleration policy. Retrieved from Institute for Research and Policy on

Acceleration website: http://www.accelerationinstitute.org/Resources/Policy_Guidelines/


19/27

Krippendorff, K., (1980). Content analysis: An introduction to its methodology. Thousand Oaks,

CA: SAGE.

Landrum, M. S., & Shakelee, B. D. (Eds.). (1998). The NAGC PreK-12 Gifted Program

Standards. Washington, DC: NAGC.

Loius, K. S., Thomas, E., Gordon, M. F., Febey, K. S., (2008). State leadership for school

improvement: An analysis of three states.Educational Administration Quarterly, 44,

562-592.

Lynn, L. E., Jr. (1980). Crafting policy analysis for decision makers [Interview with Laurence E.

Lynn, Jr.].Educational Evaluation and Policy Analysis, 2, 85-90.

National Association for Gifted Children and the Council of State Directors of Programs for the

Gifted. (2009). 2008-2009 State of the states: A report by the National Association for

Gifted Children and the Council of State Directors of Programs. Washington, DC:

Author.

Mark, M. M., Cooksy, L. J., & Trochim, W.M.K. (2009). Evaluation policy: An introduction and

overview. In W.M.K. Trochim, M. M. Mark, & L. J. Cooksy (Eds.).Evaluation policy

and evaluation practice. New Directions for Evaluation, 123, 3-11.

McDonnell, L. M., & Elmore, R. F., (1987). Getting the job done: Alternative policy

instruments.Educational Evaluation and Policy Analysis, 9, 133-152.

Merriam, S. B. (2009). Qualitative research: A guide to design and implementation (3rd ed.). San

Francisco: CA: Jossey-Bass.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook

(2nd ed.). Thousand Oaks, CA: SAGE.

Park, B. N. (Ed.). (1984). Evaluating programs for the gifted and talented [Special issue].

Journal for the Education of the Gifted. 7(1).


20/27

Passow, A. H., & Rudnitski, R. A., (1993). State policies regarding education of the gifted as

reflected in legislation and regulation. (Report No. CRS93302). Retrieved from the

National Research Center on the Gifted and Talented website:


Patton, M. Q., (2003). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks,

CA: SAGE.

Patton, M. Q., (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: SAGE.

Preskill, H., (2008). Evaluations second act: A spotlight on learning. American Journal of

Evaluation, 29, 127-138.

Renzulli, J. S. (1975).A guidebook for evaluating programs for the gifted and talented. Venture,

CA: Office of the Ventura County Superintendent of Schools.

Segerholm, C. (2003). Researching evaluation in national (state) politics and administration: A

critical approach.American Journal of Evaluation, 24, 353-372. doi:

10.1177/109821400302400305

Stambaugh, T., Worley, B., VanTassel-Baska, J., & Brown, E. (2005). Analysis of state gifted

education policy self-assessment forms: Results from the NAGC state policy workshop,

November, 2004. Washington, DC: National Association for Gifted Children.

Stepanek, J., and Peixotto, K. (2009). Models of response to intervention in the Northwest

Region states (Issues & Answers Re port, REL 2009No. 079). Washington, DC: U.S.

Department of Education, Institute of Education Sciences, National Center for Education

Evaluation and Regional Assistance, Regional Educational Laboratory Northwest.

Retrieved from www.ies. ed.gov/ncee/edlabs.


21/27

Swanson, M. (2002).National survey on the state governance of K-12 gifted and talented

education. Summary report. Nashville: Tennessee Initiative for Gifted Education Reform.

(ERIC Document Reproduction Service No. ED471886)

Trochim, W.M.K. (2009). Evaluation policy and evaluation practice. In W.M.K. Trochim, M. M.

Mark, & L. J. Cooksy (Eds.).Evaluation policy and evaluation practice. New Directions

for Evaluation, 123, 13-32.

Trochim, W. M. K., Mark, M. M., & Cooksy, L. J., (Eds.). (2009). Evaluation policy and

evaluation practice.New Directions in Evaluation, 123.

VanTassel-Baska, J. (2004). Metaevaluation findings: A call for gifted program quality. In J.

VanTassel-Baska & A. X. Feng (Eds.),Designing and utilizing evaluation for gifted

program improvement(pp. 227-245). Waco, TX: Prufrock Press.

VanTassel-Baska, J. (2006a). A content analysis of evaluation findings across 20 gifted

programs: A clarion call for enhanced gifted program development. Gifted Child

Quarterly, 50, 199-215.

VanTassel-Baska, J. (2006b). State policies in gifted education. In J. H. Purcell & Eckert, R. D.,

(Eds.).Designing services and programs for high-ability learners: A guidebook for gifted

education. Thousand Oaks, CA: Corwin Press & The National Association of Gifted

Children.

VanTassel-Baska, J., & Feng, A. X. (Eds.). (2004). Designing and utilizing evaluation for gifted

program improvement. Waco, TX: Prufrock Press.

Yin, R. K. (2009). Case study research: Design and methods (4th ed.). Thousand Oaks, CA:

SAGE.


22/27

Appendix A

Quality Indicators Matrix

Table X:

Cross-Case Analysis of Quality Criteria

Key: Not observed

Criteria Not Met

Criteria Partial Met

Criteria Met

Case

A

Case

B

Case

C

Case

D

Case

E

I

n

di

c

at

o

rs

of

H

ig

h-

Q

u

alit

y

P

ol

ic

y

(

V

a

n

Tas

se

l-

B

as

k

a,

Clarity

C1 Inclusion of clear,

unambiguous language

C2 Promotes common

interpretation

C3 Easy determination of

compliance by LEAs

C4 Easy determination of actions

necessary for compliance

Connectedness

C5 Internal validity with other

policies in gifted education

C6 Compatibility with other

policies in gifted education

C7 Linkage with general education

policies

Feasibility for

Implementation

C8 Practicality of implementation

at local level

C9 Replicability in other states

Research-Based C10 Grounding in research-basedbest practices

C11 Congruency with new research

in gifted education


23/27

2

0

0

4)

N

A

G

C

P

reK

-

1

2

P

r

o

g

r

a

m

Sta

n

d

a

r

d

s

fo

r

P

r

o

g

r

a

m

E

v

al

u

at

io

n

(1

9

9

8)

Purposeful EvaluationC12 Information reflects needs of

constituency groups

Efficient and

Economical Evaluation

C13 Sufficient time, money, and

personnel

Competent and Ethical

Evaluation

C14 Competent, knowledgeableevaluator

C15 Evaluation design

C16 Instruments and procedures

C17 Formative and summative

designs

C18 Confidentiality and confidence

of participants

Evaluation Report

C19 Written report content and

design


24/27

Appendix B

Survey Items

Instructions:

We are investigating the different ways that state policies influence the practice of program

evaluation in gifted and talented programs. You have been selected to receive this questionnaire

because information gleaned from the recent State of the States (2009) report indicates that your

state has some type of policy that may influence gifted program evaluation (e.g., self-evaluation,compliance monitoring, approval of local plans). We are conducting case studies of different

types of policies and would like your state to serve as a illustrative case for one type of policy.

If you are interested in participating in this study, please complete the answer the following

questions. Upon receipt of your completed questionnaire, you will be contacted to schedule a 30-

minute follow-up interview by phone.

Definitions: Throughout this document I refer to the evaluation system. By this I mean any

process, mandate, or initiative that requires gifted and talented programs to conduct, or supply

information verifying that they have conducted, a program evaluation.

Survey Questions

1. Briefly describe the system that your state uses to encourage, influence, or mandate gifted

and talented programs to conduct program evaluations.

2. Who is responsible for overseeing the implementation of this evaluation system? What

role do they play in the oversight of this evaluation system?

3. We have collected the following documents from your state website as documents that

reflect the policies influencing gifted program evaluation in your state. Are there any

other documents that would be valuable for our study? If so, how might we receive a

copy of those items?

4. If available, please also provide any other documents that would demonstrate your states

evaluation system in action (e.g., a typical evaluation report from a local gifted, an on-site

monitoring report).

5. In the future we may conduct a follow-up study that explores various stakeholders

experiences with this evaluation system. Please provide names and contact information

for stakeholders in your state that could share their experiences with the evaluation

system. These might be regional technical assistance providers, reviewers of plans, gifted

program coordinators who have gone through the process, or on-site monitors.


25/27

Appendix C

Follow-up Interview Questions

Thank you for taking the time to talk with me today about your states policy for gifted program

evaluation. I have several questions that I would like to ask you about your states policy. Our

conversation should take about 25 to 30 minutes.

1. I recently sent you a description that Ive written to characterize the nature of your states

policy for the evaluation of gifted programs. Have I accurately described your states

evaluation policy? Are there any points that you would like to clarify for me?

2. What do you think this policy is achieving?

3. What would be an ideal result of this policy from the states perspective? Do you feel that

your state is achieving your ideal results? (Probe intended uses and intended users)

4. How do you use the information that is gathered through this evaluation system?

5. How are local gifted programs affected by this policy? What role do you think this policy

plays in the quality of local gifted programs?

6. What have been some of the successes of this policy? Challenges?

7. What changes would you make to this evaluation system if it were within your power?

8. What advice would you offer to a policymaker who was considering a similar system of

gifted program evaluation in his or her own state?


26/27

Appendix D

Sample Cross-Case Data Displays

Table D1:

State Evaluation Policy Types in Gifted Education with Intended Uses and Intended

Users

Evaluation

Policy Type Description

States

Using this

Policy

Type

Intended Uses

of Evaluation

Results

Intended Users

of Evaluation

Results

Type A

Type B

Type C

Type D

Type E

Key

Type A

Type D

Type B

Type E

Type C


27/27

Figure D1. Visual display of policy type by state. (Note: Those states in black represent the

states in Table X of the proposal.

Date post:	30-May-2018
Category:	Documents
Upload:	api-20487015
View:	214 times
Download:	0 times

State Evaluation Policies in Gifted Education: Examining Options for Policymakers

Documents