Teacher Education Department Unit Assessment Report 2005-2006

8/14/2019 Teacher Education Department Unit Assessment Report 2005-2006

1/53


2/53

Unit-wide assessments of its students in meeting the primary and shared goals of all School of

Education programs. The data and analyses included in this report include all programs and

students in the Teacher Education Department and will, once procedures have been validated inall School of Education programs, be available for aggregation into a School-wide report. In

the meantime, the Department of Teacher Education is using this new assessment procedure forits own, departmental self-evaluation, and that is the purpose of the activities and analyses

reported below2.

Because this was the first year in which this data was collected, results (especially for the fall

2005 semester) are incomplete, but they nonetheless suggest that we are meeting our principalgoals. Most importantly for this initial assessment, they demonstrate good levels of reliability

and validity for our rating system, which appears to be sufficiently robust to allow meaningful

interpretation of data. This years effort was designed to test the system and to provide

2 To further clarify for those not familiar with the process we have recently adopted, the Unit

Assessment activities described in this report are just the newest aspect of our self-evaluationprocess. The evidence reviewed by the NCATE team of examiners was based on a variety of

other quantitative internal criteria (e.g., minimum grade requirements in education courses and

minimum GPA requirements) and external criteria (e.g., scores on a variety of national teachercertification tests, scores on national tests of competence in fundamental skills such as writing

and mathematics, and scores on national tests of content knowledge in the areas of science,

history, geography, mathematics, literature, and the arts). In these areas, Riders Teacher

Preparation Program has set GPA and PRAXIS testing requirements that meet (and in severalareas exceed) state certification requirements.

We also have demonstrated that we meet or exceed the dozens of program-specific

criteria set by the many Specialty Program Associations (SPAs) affiliated with NCATE. These

include the national groups that NCATE trusts to evaluate elementary education programs, earlychildhood education programs, special education programs, and secondary subject-area teaching

programs like mathematics education and English/language arts education. Each of these SPAssets very specific, performance-based requirements, and our program presented (as part of the

NCATE re-certification process) detailed evidence of our success in meeting these standards.

This evidence was evaluated both by the SPAs and by NCATE as part of our recent (and

successful) evaluation for re-certification. We will continue to collected this data for futurereports to the individual SPAs. The Unit Assessment procedures that are the subject of this

report are an addition to our assessment procedures but do not replace or negate those other,

program-specific evaluations.The Unit Assessment described here was initially designed under NCATE guidance in the

spring of 2005 as a part of that re-certification process and was further developed by the TeacherEducation Department in the summer and fall of 2005. Its goal is to assess how well our TeacherPreparation Program is meeting the overarching and unifying goals of our program -- the goals

that all our various programs share. These goals, which are set forth in Riders School of

Educations Conceptual Framework, commit us to fostering committed, knowledgeable,reflective professionals. It is therefore student performance in those four areas -- commitment,

knowledge, reflection, and professionalism -- that we are assessing.

Teacher Education Unit Assessment, 6-06, Part I (Introduction) 2


3/53

direction for ways to improve and refine our data collection and analysis. Our two goals were:

1. to assess the reliability and validity of the Unit Assessment procedures as applied in theTeacher Preparation Program

2. to assess how well our students are doing in the four areas -- commitment, knowledge,reflection, and professionalism -- that are the primary shared goals of all School of

Education programs, as outlined in our Conceptual Framework.

General Plan for the Unit Assessment

The School of Education at Rider University strives to prepare students for educational settings

by fostering committed, knowledgeable, reflective professionals. Our plan is to assess

candidates' development in these four areas (commitment, knowledge, reflection, and

professionalism) at three points: upon matriculation (following completion of the first two

required courses in the department); upon completion of methods courses; and uponcompletion of student teaching3. These assessments are not part of grading students in their

coursework and will not routinely be reported to students. The ratings are done with the primarygoals of program evaluation and alerting the program of the need for intervention where

necessary with students who are making unacceptable progress. These assessments are part of a

larger School of Education Unit Assessment Plan.

We will conduct reliability and validity studies at regular intervals to document that the data

being collected are meaningful. This is the report of the first such validity study. In all cases,

two or more independent assessments of each student will be made at each level. These will bemade by the professors of EDU-106 and EDU-206 at the first level (matriculation), professors

of all required methods courses at the next level (these courses vary depending on the areas in

which students hope to be certified), and the student teaching supervisor, cooperating teacher,and seminar leader at the final (student teaching) level.

PART II: FALL 2005 SEMESTER DATA, ANALYSIS, AND CONCLUSIONS4

Fall 2005 Results

A total of 488 ratings are included in this analysis. Of these, 144 were of students in

sophomore-level courses (level 1 -- matriculation) and 320 were students in junior-level

3 These three levels are being used throughout the School of Education, including GraduateEducation programs, and the data may in the future be aggregated for an overall Unit

Assessment. No such aggregation of data will be attempted until each program can demonstrate

reliability and validity in its assessments, however.4 The data and analyses reported in this section were previously reported, in slightly different

form, in February 2006.



4/53

courses (level 2 -- methods courses)5. No ratings were collected for level 3 students (at the

conclusion of student teaching) in this initial testing of the assessment system.

Of the level 1 ratings, 67 were from EDU-106 and 84 were from EDU-206. The breakdown forlevel 2 ratings was as follows:

Number of Level 2 Ratings by Course

Course Number of Ratings

ELD-307 57

ELD-308 38

ELD-375 58

ELD-376 42

SED-400 11SED-405 18

SED-410 1

SED-415 9

SED-420 5

SED-431 17

SPE-2016 23

SPE-301 28

5 A total of 24 student ratings were excluded from analyses based on level (level 1 =

matriculation, or sophomore level courses, and level 2 = methods courses, or junior-levelcourses), but they were included in the total ratings. These 24 ratings were for students in the

following courses: ECE 322 (8 students), ECE 440 (5 students), ECED 522 (2 students -- these

were graduate students who were mistakenly included in the data collection), ECED 540 (5students -- these were graduate students who were mistakenly included in the data collection),

and EDU 320 (4 students). These ratings were excluded from level 1 and level 2 analysesbecause it was unclear which level was appropriate, something that needs to be determined forfuture assessments. (The handful of graduate student ratings were mistakenly included because

they are in dual-listed courses.)6 The students taking this course are primarily sophomores, but because this is a methods courseit was included in this analysis as a level 2 course. This designation may be changed in the

future.



5/53

Reliability: Correlations between the ratings of all raters at each level were used to determinethe reliability of the ratings. Most students had two ratings7, and the correlations of those

ratings are reported below8.

Inter-rater Reliability -- All Ratings (N = 488)

Correlation

Commitment .810

Knowledge .775

Reflection .851

Professionalism .815

Inter-rater Reliability -- Level 1 (Sophomore Level, N = 144)

CorrelationCommitment .846

Knowledge .792

Reflection .857


Inter-rater Reliability -- Level 2 (Junior Level, N = 320)

Correlation

Commitment .846

Knowledge .767

Reflection .875


These inter-rater reliability coefficients are quite adequate for group comparisons, and althoughthey do not reach the .90 level that is optimal for making comparisons among or high-stakes

decisions about individuals, most do achieve the .80 level that is generally deemed acceptable

for individual comparisons (but, of course, the purpose and use of these assessments hasnothing to do with high-stakes decisions about individuals or comparisons among individuals).

For an inter-rater reliability measure employing a wide variety of raters but just two raters for

each student, these correlations are actually rather high (especially because different professorsare observing students in different courses and different settings). Of importance here is that

they fully meet the inter-rater reliability requirements for the purpose of group comparisons

7 Some transfer students who were taking just one course of a pair, or students who were

repeating a failed course, were the only students who received only a single rating. Those ratings

were not included in inter-rater reliability calculations.8 The All Ratings included some ratings that are not included in either the Level 1 or Level 2

ratings, as explained in footnote 2 above.



6/53

and program assessment.

Validity: The validities of these assessments were estimated by calculating correlations

between ratings and GPA. In future validity studies, other independent measure(s) of

achievement will be employed, including Education course grades and (at the final level) theevaluations students receive on the appropriate INTASC standards from their supervisors and

seminar leaders.

For our initial assessment we limited our analysis to an investigation of the correlations

between ratings and overall GPA prior to entering the courses in which the ratings were

conducted. In future analyses we will also look at correlations of ratings and grades in

Education courses (as noted above) and end-of-semester GPAs, but the beginning-of-semesterGPA correlations used in this analysis are helpful in demonstrating that the ratings correlate

with other, independent measures of student performance outside the School of Education.

The correlations between ratings in commitment, knowledge, reflection, and professionalism

with GPA as of the semester the students entered the education courses when the ratings were

made were as follows:

Correlations of Ratings with GPA

Correlation with GPA

Commitment .338

Knowledge .333

Reflection .335


All correlations were significant at the 0.01 level (two-tailed). This analysis, in combination

with the reliability data reported above, supports a preliminary judgment of acceptable validity of

the rating system.

Results: Ratings were done on a four-point scale: 1-Unacceptable at this level, 2-Limited

acceptability at this level, 3-Acceptable at this level, or4-Exceeds expectations at this level. Asuccessful student would therefore score 3 (Acceptable) in each domain at each level, and this

would indicate satisfactory progress through the program. Our goal, therefore, is for students

to reach this level (3) in all areas.



7/53

The mean ratings for all students, and the means by level, were as follows:

Mean Ratings

Mean Sophomore Mean Junior Mean

Commitment 3.19 3.04 3.24Knowledge 3.06 2.89 3.13

Reflection 3.08 2.94 3.13

Professionalism 3.19 3.11 3.23

Overall Means

2.95

3

3.05

3.1

3.15

3.2

Mean

Commitment

Knowledge

Reflection

Professionalis

Teacher Education Unit Assessment, 6-06, Part II (Fall 2005) 7


8/53

Sophomore- and Junior-level Means

2.7

2.8

2.9

3

3.1

3.2

3.3

Sophomore Mean Junior Mean

Commitment

Knowledge

Reflection

Professionalism

The mean rating in all categories was slightly aboveAcceptable at this level. The ratings were

higher for students at the junior level (level 2) than at the sophomore level (level 1). This wasnot predicted because ratings of levels of performance are based on levels of commitment,

knowledge, reflection, and professionalism expected at each level, and these expected levels ofperformance of course vary, with higher standards for acceptable performance at higher levels.

Although unanticipated, it is nonetheless heartening to see that these ratings do show anincrease9. At the sophomore level, commitment and professionalism reached satisfactory levels,

but knowledge and commitment were slightly below the target level of 3 (Acceptable). At the

junior level all mean ratings are significantly above theAcceptable level.

Frequency tables and graphs for each of the four domains can be found in the Fall 2005

Appendices. Overall and across domains, approximately 80% of the ratings were in theAcceptable at this levelorExceeds expectations at this level. Approximately one-fifth of the

ratings were below the acceptable level, mostly in the category ofLimited acceptability at this

level, The percentage of ratings ofUnacceptable at this levelwere approximately 2-3 percent.

As shown in the mean ratings, Level 2 students (juniors in methods courses) received generally

9 ANOVA results indicated statistically significant interaction effects of level and domain(commitment, knowledge, reflection, and professionalism) for three of the domains (commitment

[p = .024], knowledge [p = .009], and reflection [p = .050]) at the .05 level.



9/53

higher ratings than Level 2 students (sophomores in their first education courses). The

percentage of ratings in theAcceptable at this levelorExceeds expectations at this levelforLevel 2 students ranged from 82.5% to 85%. Having 15 to 17.5% of ratings of juniors below

the Acceptable level, while an improvement from the Level 1 ratings, is certainly cause for

concern. In the Level 3 ratings it is to be hoped that fewer students will be rated less thanAcceptable in any of the four domains. The low level of Unacceptable ratings, however (ranging

from 1.6% to 2.8% -- less than 10 students in all) is encouraging, however. Those students will

need to repeat courses or develop a program to improve in areas in which they are deficientbefore they can be successful as student teachers. Given that Unacceptable ratings were as high

as 6.3 percent among Level 1 students, it is clear that the numbers of these most troublesome

ratings is decreasing as students progress through the program. The level for juniors (Level 2) is

low -- possibly as low as is practicably possible -- and it is hoped that the level will be evenlower when Level 3 (student teacher) ratings are collected in Spring 2006.

Also of interest is the fact that ratings in these four domains (commitment, knowledge,

reflection, and professionalism) are highly intercorrelated, as one would expect. These areclearly not orthogonal variables, but rather interdependent attributes that are all necessary for

successful teaching. While conceptually it is easy to distinguish commitment, knowledge,reflection, and professionalism, in practice they are not independent and will both overlap and

predict one another. For example, a student who is committed to teacher preparation is likely

to be more knowledgeable, professional, and reflective than a student lacking suchcommitment. In this sense these four scales can be thought of as four parts of a single scale,

rather the way different methods of assessment in a course (tests of various kinds, papers,

presentations, etc., while on the surface quite different, are essentially measuring the same

construct (understanding of course content) in different ways. The correlation matrix appearsbelow10.

10 If viewed as a single test, Cronbachs alpha for these ratings would be 0.931. The inter-raterreliability estimates are only slightly higher than the cross-domain correlations, which suggests

that these four areas might be thought of as inter-related parts of a whole rather than as

independent constructs. They certainly tend to go together, but this does not mean that theycannot be understood as separate constructs. By way of analogy, skill in multiplying fractions is

likely to be very highly correlated with skill in dividing fractions, and knowledge of appropriateuse of quotation marks is likely to be highly correlated with skill in capitalization, but in bothcases the two paired skills are conceptually quite different (and though it would be unusual, one

could have very different levels of expertise in, say, use of quotation marks, and capitalization).

Similarly, although one would expect that commitment, knowledge, reflection, andprofessionalism would be highly intercorrelated, that does not mean that they represent a single

construct.



10/53


11/53

demonstrating acceptable levels of commitment, knowledge, reflection, and professionalism.

Future Directions and Recommendations: The following additions and changes will

improve our Unit Assessment Plan:

1. Expand data collection to include level 3 (student teachers)2. For level 3 data collection, add INTASC Standards assessment data.

3. Decide how to use SPE and ECE data (and be sure to exclude ratings of graduate

students)4. Decide which separate program analyses to run (if any), such as Elementary Education,

Secondary Education, etc.

5. Needed analyses:

Overall means of Commitment, Knowledge, Reflection, & Professionalism

ratings and means for Commitment, Knowledge, Reflection, & Professionalism

ratings at each level (1, 2 & 3)

Overall distribution of Commitment, Knowledge, Reflection, & Professionalism

ratings and distribution of Commitment, Knowledge, Reflection, &Professionalism ratings at each level (1, 2 & 3)

Inter-rater reliability for Commitment, Knowledge, Reflection, & Professionalism

ratings (2 raters for most students at levels 1 and 2, 3 raters for students at level 3)

Means of grades in current Education courses

Correlations among GPA (at end of the semester), means of grades in currentEducation courses, and Commitment, Knowledge, Reflection, & Professionalism

ratings -- both overall and by level

For level 3 only, correlations among 11 INTASC ratings and Commitment,Knowledge, Reflection, & Professionalism ratings

ANOVA of mean differences among the 3 levels for Commitment, Knowledge,

Reflection, & Professionalism ratings



12/53

Fall 2005 Appendices

Frequency Tables and Graphs --All Ratings

Commitment - All Ratings

Rating Frequency Percent Cumulativepercent

1-Unacceptable

at this level

10 2.0 2.0

2-Limited acceptability

at this level

87 17.8 19.9

3-Acceptable

at this level

193 39.5 59.4

4-Exceeds expectations

at this level

198 40.6 100.0

Total 488 100.0


0

50

100

150

200

Frequency

Unacceptable

Limited acceptability

Acceptable

Exceeds expectation



13/53

Knowledge - All Ratings

Rating Frequency Percent Cumulative

percent

1-Unacceptableat this level

15 3.1 3.1


at this level

98 17.8 23.2

3-Acceptable

at this level

218 39.5 67.8


at this level

157 40.6 100.0

Total 488 100.0


0

50

100

150

200

250

Frequency

Unacceptable

Limited acceptabilityAcceptable

Exceeds expectation



14/53

Reflection - All Ratings


percent


12 2.5 2.5


at this level

93 19.1 21.5

3-Acceptable

at this level

229 46.9 68.4


at this level

154 31.6 100.0

Total 488 100.0


0

50

100

150

200

250

Frequency

Unacceptable


Acceptable

Exceeds expectation



15/53

Professionalism - All Ratings


percent


15 3.1 3.1


at this level

73 15.0 18.0

3-Acceptable

at this level

204 41.8 59.8


at this level

196 40.2 100.0

Total 488 100.0


0

50

100

150

200

250

Frequency

Unacceptable


Acceptable

Exceeds expectation



16/53

Frequency Tables and Graphs --Level 1 (Sophomore)

Commitment - Level 1 (Sophomore)


1-Unacceptable

at this level

3 2.1 2.1


at this level

35 24.3 26.4

3-Acceptableat this level

59 41.0 67.4

4-Exceeds expectationsat this level

47 32.6 100.0

Total 144 100.0

Commitment - Level 1 (Sophomores)

0

10

20

30

40

50

60

Frequency

Unacceptable


Acceptable

Exceeds expectation



17/53

Knowledge - Level 1 (Sophomore)


percent1-Unacceptable

at this level9 6.3 6.3

2-Limited acceptabilityat this level

44 30.6 36.8

3-Acceptable

at this level

45 31.3 68.1


at this level

46 31.9 100.0

Total 144 100.0

Knowledge - Level 1 (Sophomores)

0

10

20

30

40

50

Frequency

Unacceptable

Limited acceptabilityAcceptable

Exceeds expectation



18/53

Reflection - Level 1 (Sophomore)


percent


7 4.9 4.9


at this level

38 26.4 22.2

3-Acceptable

at this level

55 38.2 65.5


at this level

44 30.6 100.0

Total 144 100.0

Reflection - Level 1 (Sophomores)

0

10

20

30

40

50

60

Frequency

Unacceptable

Limited acceptabilit

Acceptable

Exceeds

expectations



19/53

Professionalism - Level 1 (Sophomore)


percent


6 4.2 4.2


at this level

26 18.1 22.2

3-Acceptable

at this level

58 40.3 62.5


at this level

54 37.5 100.0

Total 144 100.0

Professionalism - Level 1 (Sophomores)

0

10

20

30

40

50

60

Frequency

Unacceptable


Acceptable

Exceeds

expectations



20/53

Frequency Tables and Graphs --Level 2 (Junior)

Commitment - Level 2 (Junior)


1-Unacceptable

at this level

7 2.2 2.2


at this level

49 15.3 17.5


125 39.1 56.6


139 43.4 100.0

Total 320 100.0

Commitment - Level 2 (Juniors)

0

20

40

60

80

100

120

140

Frequency

Unacceptable


Acceptable

Exceeds

expectations



21/53

Knowledge - Level 2 (Junior)


percent1-Unacceptable

at this level6 1.9 1.9


45 14.1 15.9

3-Acceptable

at this level

169 52.8 68.8


at this level

100 31.3 100.0

Total 320 100.0

Knowledge - Level 2 (Juniors)

0

20

40

60

80

100

120

140

160

180

Frequency

Unacceptable


Acceptable

Exceeds

expectations



22/53

Reflection - Level 2 (Junior)


percent


5 1.6 1.6


at this level

46 14.4 15.9

3-Acceptable

at this level

170 53.1 69.1


at this level

99 30.9 100.0

Total 320 100.0

Reflection - Level 2 (Juniors)

0

20

40

60

80

100

120

140

160

180

Frequency

Unacceptable


Acceptable

Exceeds

expectations



23/53

Professionalism - Level 2 (Junior)


percent


9 2.8 2.8


at this level

39 12.2 15.0

3-Acceptable

at this level

141 44.1 59.1


at this level

131 40.9 100.0

Total 320 100.0

Professionalism - Level 2 (Juniors)

0

20

40

60

80

100

120

140

160

Frequency

Unacceptable


Acceptable

Exceeds

expectations



24/53

PART III: SPRING 2006 SEMESTER DATA, ANALYSIS, AND CONCLUSIONS

Spring 2006 Reliability and Validity Studies

A total of 503 ratings are included in this analysis. Of these, 90 were of students insophomore-level courses (level 1 -- matriculation), 256 were students in junior-level courses

(level 2 -- methods courses), and 157 were students in senior-level courses (level 3 -- student

teaching).

Reliability:

Correlations between the ratings of all raters at each level were used to determine the reliability

of the ratings. Most students at Levels 1 and 2 had two independent ratings by twoprofessors13. Level 3 students had 3 independent ratings by their seminar leader, their

cooperating teacher, and their student teaching supervisor. The paired inter-rater reliability

correlations of those ratings are reported below.

Inter-rater Reliability -- All Ratings

Correlation

Commitment .795

Knowledge .762

Reflection .826


Inter-rater Reliability -- Level 1 (Sophomore Level,)

Correlation

Commitment .745

Knowledge .776

Reflection .818


Inter-rater Reliability -- Level 2 (Junior Level)

Correlation

Commitment .762

Knowledge .741

Reflection .808Professionalism .816

13 Some transfer students who were taking just one course of a pair, or students who wererepeating a failed course, were the only students who received only a single rating. Those ratings

were not included in inter-rater reliability calculations.

Teacher Education Unit Assessment, 6-06, Part III (Spring 2006) 24


25/53

Inter-rater Reliability -- Level 3 (Senior/Student Teaching Level,)

Correlation

Commitment .883

Knowledge .749

Reflection .833


These inter-rater reliability coefficients are quite adequate for group comparisons, for which .60 or higher is generally recommended. For an inter-rater reliability measure using just two

raters, these correlations are actually rather high (especially because different professors are

observing students in different courses and different settings). Of importance here is that theyfully meet the inter-rater reliability requirements for the purpose of group comparisons and

program assessment. It is also interesting to note that the highest inter-rater reliabilities were

at Level 3, the ratings of student teachers, which is arguably the most important assessment ofthe three because the Level 3 assessments are made as students are completing the TeacherPreparation Program. These inter-rater reliabilities of Level 3 students are so good that they

actually reach reliability levels sufficient for use in making individual comparisons among

students or decisions about individual students (in which a .90 level is desirable but .80 isgenerally considered quite acceptable), although that was not their purpose.

Validity: The validities of these assessments were estimated by calculating correlations

between ratings in commitment, knowledge, reflection, and professionalism and the mean grade

in current education courses (except for Level 3 students, for which a more detailed analysis was

possible because students also had ratings on each of 11 INTASC Standards). An additionalcorrelation between ratings in commitment, knowledge, reflection, and professionalism and GPA

at the beginning of the semester (which does not include grades in the courses given by the

raters) is also reported. For Level 3 students, other independent measure(s) of achievement --the evaluations students have received on the appropriate INTASC standards from their

supervisors and seminar leaders -- are also correlated with ratings in commitment, knowledge,

reflection, and professionalism.

Overall (all levels combined)

Correlations of ratings with Education course grades: The overall correlations for all levels

between ratings in commitment, knowledge, reflection, and professionalism with course gradesin current education courses were as follows (all statistically significant at the 0.01 level [2-

tailed]):



26/53

Overall Correlations of Ratings with Education Course Grades

Correlation with Grades

Commitment .488

Knowledge .569


Correlations of ratings with GPA: The overall correlations between ratings in commitment,knowledge, reflection, and professionalism with GPA as of the semester the students entered the

education courses when the ratings were made were as follows (all statistically significant at the

0.01 level [2-tailed]):

Overall Correlations of Ratings with GPA


Commitment .295Knowledge .325

Reflection .342


Level 1

Correlations of ratings with Education course grades: For Level 1 students the correlations

between ratings in commitment, knowledge, reflection, and professionalism with course gradesin current education courses were as follows (all statistically significant at the 0.01 level [2-

tailed]):

Level 1 Correlations of Ratings with Education Course Grades

Correlation with Grades

Commitment .455

Knowledge .646

Reflection .545


Correlations of ratings with GPA: For Level 1 students the correlations between ratings in

commitment, knowledge, reflection, and professionalism with GPA as of the semester the

students entered the education courses when the ratings were made were as follows (allstatistically significant at the 0.05 level [2-tailed] except Commitment, which was significant at

the 0.01 level [2-tailed]):



27/53

Level 1 Correlations of Ratings with GPA


Commitment .306

Knowledge .240


Level 2

Correlations of ratings with Education course grades: For Level 2 students the correlations

between ratings in commitment, knowledge, reflection, and professionalism with course grades

in current education courses were as follows (all statistically significant at the 0.01 level [2-tailed]):

Level 2 Correlations of Ratings with Education Course Grades

Correlation with GradesCommitment .518

Knowledge .599

Reflection .459



commitment, knowledge, reflection, and professionalism with GPA as of the semester the

students entered the education courses when the ratings were made were as follows (allstatistically significant at the 0.01 level [2-tailed]:



Commitment .302

Knowledge .321

Reflection .347


Level 3

Correlations of ratings with separate ratings on INTASC Standard criteria: For Level 3

students the correlations between ratings in commitment, knowledge, reflection, andprofessionalism with ratings in the 10 INTASC Standards plus an 11th Standard of how well

student teachers help students develop thinking and problem solving skills (which was added at

the suggestion of the NCATE Accreditation Team) were as follows (all statistically significant atthe 0.01 level [2-tailed]:



28/53

STANDARD Commitment Knowledge Reflection Professional

Principle 1, Understands

Content: The teacher

understands the central concepts,tools of inquiry, and structures ofthe discipline(s) he/she teaches

and can create learning

experiences that make these

aspects of subject mattermeaningful for students.

.514 .556 .434 .453

Principle 2, Understands

Development: The teacher

understands how children learn

and develop, and can provide

learning opportunities thatsupport their intellectual, social,

and personal development.

.531 .489 .486 .493

Principle 3, UnderstandsDifference: The teacher

understands how students differin their approaches to learning

and creates instructional

opportunities that are adapted tothe diverse learner.

.511 .447 .442 .471

Principle 4, Designs

Instructional Strategies: Theteacher understands and uses a

variety of instructional strategiesto encourage students

development of critical thinking,

problem solving, and

performance skills.

.605 .578 .558 .554

Principle 5, Manages and

Motivates: The teacher uses anunderstanding of individual and

group motivation and behavior to

create a learning environmentthat encourages positive social

interaction, active engagement in

learning and self-motivation.

.505 .397 .451 .471

Principle 6, Communicates:

The teacher uses knowledge of

effective verbal, nonverbal and

.495 .452 .437 .424



29/53

media communicationtechniques to foster active

inquiry, collaboration, and

supportive interaction in the

classroom.Principle 7, Plans and

Integrates:: The teacher plans

instruction based upon

knowledge of subject matter,

students, the community, andcurriculum goals.

.616 .534 .578 .537

Principle 8, Evaluates: Theteacher understands and uses

formal and informal assessment

strategies to evaluate and ensure

the continuous intellectual,social, and physical development

of the learner.

.512 .465 .430 .431

Principle 9, Reflects onPractice: The teacher is a

reflective practitioner whocontinually evaluates the effects

of his/her choices and actions on

others (students, parents, andother professionals in the

learning community) and who

actively seeks out opportunitiesto grow professionally.

.617 .542 ..598 .560

Principle 10, Participates in the

Professional Community: The

teacher fosters relationships with

school colleagues, parents, and

agencies in the larger communityto support students learning and

well-being.

.390 .269 .381 .317

Principle 11. Develops Thinking

and Problem-Solving Skills:

The teacher is able to design andimplement lessons that foster the

growth of students' critical

thinking and problem-solving

abilities.

.501 .466 .457 .482



30/53


commitment, knowledge, reflection, and professionalism with GPA as of the semester thestudents entered the education courses when the ratings were made were as follows (all

statistically significant at the 0.01 level [2-tailed]:



Commitment .346

Knowledge .372

Reflection .327


Every one of the many predicted correlations was statistically significant at the 0.05 level )two-

tailed), and all but a handful of these were significant at the 0.01 level (2-tailed). The results are

therefore so consistent and so conclusive that a narrative description of these scores of datapoints would be superfluous, but complete data are presented above of all observed correlations

for verification.

Conclusions Regarding Reliability and Validity from Spring 2006 Assessment:

As was found in the initial fall 2005 semester evaluation of this rating and Unit Assessment

system, these results suggest that our proposed model of assessment based on our Conceptual

Framework is viable, reliable, and valid. Both overall and at each level the ratings in the areasof commitment, knowledge, reflection, and professionalism have proven to be highly valid (as

well as reliable, as demonstrated in the previous section). The Teacher Education Departmentthere concludes that these ratings can be reported and used to judge the effectiveness of theundergraduate Teacher Preparation Program at Rider University.



31/53

Results of the Spring 2006 Assessment:

Ratings were done on a four-point scale: 1-Unacceptable at this level, 2-Limited acceptability

at this level, 3-Acceptable at this level, or4-Exceeds expectations at this level. A successful

student would therefore score 3 (Acceptable) in each domain at each level, and this wouldindicate satisfactory progress through the program. Our goal, therefore, is for students to reach

this level (3) in all areas.

The mean ratings for all students, and the means by level, were as follows:

Mean Ratings

Overall

Mean

Level I:Sophomore

Mean

Level 2:Junior

Mean

Level 3:

Senior

(StudentTeacher)

MeanCommitment 3.43 3.24 3.32 3.60

Knowledge 3.24 3.23 3.06 3.42

Reflection 3.27 3.21 3.07 3.55

Professionalism 3.38 3.11 3.23 3.61

These results are presented graphically on the following pages.



32/53

Overall Means for All Students

3.1

3.15

3.2

3.25

3.3

3.353.4

3.45

Overall Mean

Commitment

Knowledge

Reflection

Professionalis

Means for Level 3 Students

3.3

3.35

3.4

3.453.5

3.55

3.6

3.65

Level 3: Senior

(Student Teacher)

Mean

Commitment

Knowledge

Reflection

Professional



33/53


2.9

3

3.1

3.2

3.3

3.4

Level 2: Junior Mean

Commitment

Knowledge

Reflection

Professional


3

3.05

3.1

3.15

3.2

3.25

Level I: Sophomore

Mean

CommitmentKnowledge

Reflection

Professional



34/53

Means by Level for Commitment

3

3.1

3.2

3.3

3.4

3.5

3.6

All I 2 3

Commitmen



35/53

Means by Level for Knowledge

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

All I 2 3

Knowledge



36/53

Means by Level for Reflection

2.8

2.9

3

3.1

3.2

3.3

3.4

3.53.6

All I 2 3

Reflection



37/53

Means by Level for Professionalism

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

3.6

3.7

All I 2 3

Professional



38/53

As was observed in the Fall 2005 Assessment, the mean rating in all categories was above the

target level,Acceptable at this level. For Level 3 students, the mean ratings was closer toExceeds expectations at this levelin every area except knowledge.

Frequency tables and graphs for each of the four domains can be found in the Spring 2006Appendices below. Overall and across domains, approximately 80% of the ratings were in the

Acceptable at this levelorExceeds expectations at this level. Approximately one-fifth of the

ratings were below the acceptable level, mostly in the category ofLimited acceptability at this

level, The percentage of ratings ofUnacceptable at this levelwere approximately 1 percent.

This low level of Unacceptable ratings (ranging from 0.5% for Reflection to 1.4% for

Professionalism) is encouraging. Those students will need to repeat courses or develop aprogram to improve in areas in which they are deficient before they can be successful as student

teachers, or they may simply be asked to leave the program. Of those Unacceptable ratings, only

a single one came from Level 3 -- the Student Teaching level. This is heartening. Whether this

is a result of improvement prior to student teaching or a weeding out of unacceptable candidatesprior to student teaching cannot be ascertained from this data, but the important point is that,

with a single exception (one rating of Unacceptable in the area of Commitment), all studentteachers received three ratings of at leastLimited acceptability at this levelin all areas14.

Of course,Limited acceptability at this levelis not adequate, and the Teacher EducationDepartment needs to strive to even further limit the numbers of students performing at this

level. There were an average of 11 such ratings (of 157 total) in this category in each of the

four areas rated. A total of 7.166% of the ratings of student teachers therefore fell into either

the Unacceptable at this level(1 rating out of 628, or 0.0016%) orLimited acceptability at this

level(44 out of 628, or 0.0701%). On the positive side, this means that approximately 93% of

all ratings were in either the Acceptable at the level or Exceeds expectations at this levelcategories.

Once again, as in the Fall 2005 Study, ratings in these four domains (commitment, knowledge,

reflection, and professionalism) were found to be highly intercorrelated, as one would expect.Commitment, knowledge, reflection, and professionalism are interdependent attributes that are

all necessary for successful teaching. While conceptually it is easy to distinguish commitment,

knowledge, reflection, and professionalism, in practice they are not independent and will both

overlap and predict one another. For example, a student who is committed to teacherpreparation is likely to be more knowledgeable, professional, and reflective than a student

lacking such commitment. In this sense these four scales can be thought of as four parts of a

single scale, rather the way different methods of assessment in a course (tests of various kinds,

14 Careful analysis of the charts and tables in the Appendix will show that students at Level 1 in

some ways outperformed those in Level 2 in the Spring 2006 assessment, both in mean ratings

and in the area of having fewer Unacceptable ratings. This is the opposite of the results in Fall2005, but one should also note that the Level 1 cohort was unusually small in Spring 2006,

which may account for these unexpected (but not problematic) results.



39/53

papers, presentations, etc., while on the surface quite different, are essentially measuring the

same construct (understanding of course content) in different ways. The correlation matrixappears below15.

Commitment Knowledge Reflection ProfessionalismCommitment 1.00 .655 .737 .761

Knowledge .655 1.000 .733 .695

Reflection .737 .733 1.000 .750

Professionalism .761 .695 .750 1.000

Of the four areas, the one that has overall lowest ratings is the area of knowledge. This is largely

because of lower ratings in this area of Level 3 student teachers. The ratings were still quitegood -- the mean was about midway betweenAcceptable at this leveland Exceeds expectations

at this level-- but if one were to single out one area of least strength, these data suggest it

would be the area of knowledge.

15 If all four scales were combined and viewed as a single test, Cronbachs alpha for these ratings

would be 0.912.



40/53

Spring 2006 Appendices

Frequency Tables and Graphs --Ratings for All Students


Rating Frequency

1-Unacceptable

at this level

7


at this level

49


201


at this level

307


0

50

100

150

200

250

300

350 1-Unacceptableat this level

2-Limitedacceptability atthis level

3-Acceptable atthis level

4-Exceedsexpectations atthis level



41/53


Rating Frequency

1-Unacceptable

at this level

6


at this level

61

3-Acceptable

at this level

286


211


0

50

100

150

200

250

300

1-

Unacceptable

at this level

2-Limited

acceptability

at this level

3-Acceptable

at this level

4-Exceeds

expectations

at this level



42/53


Rating Frequency

1-Unacceptable

at this level

3


at this level

82

3-Acceptable

at this level

233


246


0

50

100

150

200

2501-

Unacceptable

at this level

2-Limited

acceptability a

this level

3-Acceptable

at this level

4-Exceeds

expectations

at this level



43/53


Rating Frequency

1-Unacceptable

at this level

9


at this level

48

3-Acceptable

at this level

222


285


0

50

100

150

200

250

300

1-

Unacceptabl

at this level

2-Limited

acceptability

at this level3-Acceptable

at this level

4-Exceeds

expectations

at this level



44/53

Frequency Tables and Graphs --Level 2 (Junior)

Commitment - Level 2 (Junior)

Rating Frequency1-Unacceptable

at this level6


34

3-Acceptable

at this level

130


at this level

147

Commitment - Level 2 (Juniors)

020

40

60

80

100

120

140

160

1-

Unacceptabl

at this level

2-Limited

acceptabilityat this level

3-Acceptable

at this level

4-Exceedsexpectations

at this level



45/53

Knowledge - Level 2 (Junior)

Rating Frequency


6


at this level

42

3-Acceptable

at this level

182


at this level

87

Knowledge - Level 2 (Juniors)

020

40

60

80

100120

140

160

180

200

1-

Unacceptabl

at this level

2-Limited

acceptability

at this level

3-Acceptable

at this level


at this level



46/53

Reflection - Level 2 (Junior)

Rating Frequency

1-Unacceptable

at this level

3


at this level

63

3-Acceptable

at this level

148


103

Reflection - Level 2 (Juniors)

0

20

40

60

80

100

120

140

160 1-

Unacceptabl

at this level

2-Limited

acceptability

at this level

3-Acceptable

at this level

4-Exceeds

expectations

at this level



47/53

Professionalism - Level 2 (Junior)

Rating Frequency

1-Unacceptable

at this level

8


at this level

33

3-Acceptable

at this level

159


117

Professionalism - Level 2 (Juniors)

0

20

40

60

80100

120

140

160

1-

Unacceptabl

at this level

2-Limited

acceptability

at this level

3-Acceptable

at this level


at this level



48/53

Frequency Tables and Graphs --Level 3 Senior-Student Teaching)

Commitment - Level 3 (Senior-Student Teaching)

Rating Frequency1-Unacceptable

at this level1


11

3-Acceptable

at this level

38


at this level

107

Commitment - Level 3 (Senior-Student Teaching)

0

20

40

60

80

100

1201-

Unacceptabl

at this level

2-Limited

acceptability

at this level

3-Acceptable

at this level

4-Exceeds

expectations

at this level



49/53

Knowledge - Level 3 (Senior-Student Teaching)

Rating Frequency

1-Unacceptable

at this level

0


at this level

12

3-Acceptable

at this level

67


78

Knowledge- Level 3 (Senior-Student Teaching)

0

10

20

30

40

50

60

70

80 1-Unacceptabl

at this level

2-Limited

acceptability

at this level


4-Exceeds

expectations

at this level



50/53

Reflection- Level 3 (Senior-Student Teaching)

Rating Frequency

1-Unacceptable

at this level

0


at this level

10

3-Acceptable

at this level

50


97

Reflection - Level 3 (Senior-Student Teaching)

010

20

30

40

5060

70

80

90

100

1-

Unacceptabl

at this level

2-Limited

acceptability

at this level

3-Acceptable

at this level

4-Exceeds

expectations

at this level



51/53

Professionalism - Level 3 (Senior-Student Teaching)

Rating Frequency

1-Unacceptable

at this level

0


at this level

11

3-Acceptable

at this level

39


107

Professionalism - Level 3 (Senior-Student Teaching)

0

20

40

60

80

100

120

1-

Unacceptabl

at this level

2-Limited

acceptability

at this level

3-Acceptable

at this level

4-Exceeds

expectations

at this level



52/53

PART IV: CONCLUSIONS

Regarding the reliability and validity of the Unit Assessment system, the results for the two

semesters overwhelmingly endorse the system as it has been developed and described above.There is no need to repeat these reliability and validity studies yearly (many major testing

programs do so only once a decade), but it would be prudent to perform another such study at

least every five years to ensure that the high standards achieved this year continue.

The purpose of this Unit Assessment is not to prove reliability and validity, of course. Those are

only tools that allow one to demonstrate the Unit Assessment is indeed doing what it was

designed to do -- to evaluate fairly the success of students at various levels in the undergraduateTeacher Preparation Program in the four areas that are central to our Conceptual Framework.

These evaluations suggest that, for this year, the Teach Education Department is generally

meeting its goals. This is not to say that there is not room for improvement. The one area of the

four that consistently received the lowest mean ratings is the area of knowledge. While moststudents have achieved the level ofAcceptable at this levelin the area of knowledge at all levels,

fewer have risen to the level of Exceeds expectations at this level. This is something that theDepartment of Teacher Education might set as a goal -- increasing the percentage of students

who exceed expectations in this area.

Unlike reliability and validity studies, which need be conducted only occasionally, the annual

collection and reporting of ratings of students in these key areas16 needs to be continuous for the

Teacher Education Department to continue to assess (and improve) levels of achievement of its

students. Ongoing assessment will allow the Teacher Education Department to know where it isnow and to set goals for where it hopes to be in the future. It is heartening to find that the new

Unit Assessment procedure has proven such a hardy and valid system for this kind of assessment.

16 It perhaps goes without saying (but will be said here anyway, as it was also said in theintroduction to this report) that this is not the only self-assessment system in use by theDepartment of Teacher Education and that it is not intended in any way to replace any other self-

assessment systems. Its goal is to assess student achievement in the four focal areas of

commitment, knowledge, reflection, and professionalism -- the four pillars of our ConceptualFramework -- at three key points as students pass through our Teacher Preparation Program,

and in this task it appears to have done a remarkable job.

Teacher Education Unit Assessment, 6-06, Part IV (Conclusions) 52


53/53

Date post:	30-May-2018
Category:	Documents
Upload:	scarbonaro1186
View:	216 times
Download:	0 times

Teacher Education Department Unit Assessment Report 2005-2006

Documents