eEmpowerment Zone:

An eLearning System Facilitates the Assessment of Teaching

Candidates

Overview

Provide Context Accelerate Online Program eLearning Delivery System

Exploratory Study – very much in progress Third Year Evaluation of Accelerate

Online/OPTIONS a Transition-to-Teach Program at TAMU - Results

Background

We have invested over three years of continuous effort and resources in developing a teacher preparation program that incorporates an on-line delivery system

And have come to view this delivery system as holding significant promise for the continuing education and formal development of teachers following their initial certification and entry into the profession.

Perspectives

The eLearning attributes of accessing reliable and valid multimedia resources 24/7 from a personal workstation are powerful influences for changing how professional development experiences are envisioned.

However, in the arena of eLearning, it is generally accepted that professional development efforts in public schools have not “kept pace with the rapid changes in the quality and quantity of information technology” (Denton, Davis, Strader, Durbin, & Wang, 2004; Moursund & Bielefeldt, 1999).

School policy makers wanting to improve teaching that results in greater student performance nearly always consider teacher professional development as a central component in their school improvement plans (Guskey, 2002).

Our experiences (Denton, Davis, Smith, Strader, Clark & Wang, 2004-05) with technology professional development for teacher educators and classroom teachers are consistent with the literature (Garet, Porter, Desimone, Birman & Yoon, 2001; Joyce & Showers, 2002; Loucks-Horsley, Love, Stiles, Mundry, & Hewson, 2003) regarding professional development experiences that –

emphasize academic subject matter (content), provide opportunities for “hands-on” activities (active

learning), are integrated with ongoing classroom operations

(coherence), and provide many development experiences for an

extended period of time are more likely to produce desired knowledge and

skill changes.

The Context –Accelerate Online:A Teacher Preservice Professional Development Program

Accelerate Online/OPTIONS is a three component program for certifying secondary teachers in Texas.

The Online Curriculum consists of 35 online modules developed to engage the candidate with issues identified as necessary for a beginning teacher by the Texas Board of Educator Certification .

Early Field Experience

The Field Based Experience consists of a 40 clock-hour supervised teaching field experience in a secondary school during the initial phase of the program.

Internship

The final program component is a paid internship where candidates are supported by a trained mentor and a university supervisor, who guide, observe and provide constructive feedback to the interns during their year-long development as a beginning teacher.

eEmpowerment Zone System

Comprehensive, web-based elearning, assessment and professional development system designed for diverse communities of learners (Davis, T., Director and Co-Developer).

Primary delivery system for the Accelerate Online teacher certification program in the Department of Teaching, learning, and Culture

2500 users to date, including approx.110 Accelerate Online students

http://empowermentzone.tamu.edu http://empowermentzone.tamu.edu/portfolios/center

eZone | Online Modules

The eEmpowerment Zone, is a dynamic, on-demand delivery platform developed in-house (Davis & Strader) that enables cohorts/communities of teaching candidates to access instructional modules, as well as integrated resources and tools that support teaching and learning, supervision, resource evaluation and collaboration.

eZone | Resources By organizing instructional web-based modules,

electronic portfolios, resources and tools into an integrated system, teaching candidates can seamlessly complete the online and field-components of the program, while receiving extensive support from university supervisors, mentor teachers, fellow students and program staff.

In addition, the backend eZone database houses extensive performance data on all students related to online instructional module assessments, classroom observations, and ePortfolio assessments.

Welcome VIDEO

eLearning System Design

eZone –eLearning System

Online Modules ePortfolios Classroom Observations Integrated Resources

My eAssessment Center

Cluster 1

Cluster 2

Cluster 3

Cluster 4

Module/Course Assignments

Early Field Experiences

Instructional Artifacts

Domain 1

Domain 2

Domain 3

Domain 4

Domain 5

Community Discussion Boards

Community ePortfolios

Domain 6

Domain 7

Resource Media Center

Break Room

My eAssessment Center

The My Assessment Center in the eZone and underlying database facilitates 24/7 tracking of student progress

The My eAssessment Center serves to affirm candidate program status in terms of completed assignments and module deliverables

A digital monitoring system will return the candidate to the uncompleted section of a module that was exited before the module was completed.

My eAssessment Center As a management resource for program

administrators, this digital monitoring system attends to each candidate’s visits to a module, the elapsed time spent examining the contents of the module and whether items requiring a response have in fact been completed as well as module pretest and posttest performances by the candidates.

Candidate Progress

Candidate progress in completing the modules are reviewed (weekly) to determine whether individual candidates are progressing satisfactorily in the program,

As well as determining whether modules have potential design flaws given the collective performance of the candidates on particular activities as well as overall performance on the module.

Additional Perspectives Three recent reports provide insights about alternative route

program features [Allen (2003), Seftor and Mayer (2003), and Mayer et al. (2003), cited in American Institutes for Research(2005)]

All three reports examined studies on the types and characteristics of alternative certification programs that might prove to be effective in teaching and for student learning

Allen’s literature review generated a set of principles regarding program features which he found to be indicated in successful programs

Among these five were: Solid curriculum that includes instruction in classroom basics

and teaching methods, and As much training as possible prior to teaching assignments

Research Questions

What is the relationship between candidates’ module instructional time and their performance on the Pedagogy and Professional Responsibilities (8-12) exams?

What is the relationship between candidates’ module posttest performance and their performance on the Pedagogy and Professional Responsibilities (8-12) exams?

Module Clusters for Accelerate Online Program

Module Clusters for Accelerate Online Program are organized by the following clusters:

School & Classroom Environment Designing Instruction Promoting Student Learning Professional Roles & Responsibilities Technology Applications Plus 3-4 Content Area Modules (Science,

Mathematics, History and Social Studies, English, Language Arts & Reading)

Pedagogy and Professional Responsibilities ((PPR)

Test Framework

Domain 1: Designing Instruction and Assessment to Promote Student Learning (approx. 31% of the test)

SBEC Standards Assessed PPR Standard I: The teacher designs instruction appropriate for all students that reflects

an understanding of relevant content and is based on continuous and appropriate assessment.

Online Module Alignment SBEC Standards (Knowledge and Skills) Addressed in Module

M4: Instructional Planning Knowledges 1.7K-1.11K Skills 1.6S-1.11S

M5: Performance Objectives Knowledges 1.7-1.15K Skills 1.6S-1.15S

M6: Instructional Strategies Knowledges 1.19K-1.24K 3.12K-3.14K Skills 1.19S-1.23S 3.15S-3.17S

M7: Accommodating Student Diversity in the Classroom

Knowledges 1.1K-1.6K 2.1-2.3K Skills 1.1S-1.5S 2.1-2.3S

M8: Adolescent and Learner Development Knowledges 1.1K-1.6K Skills 1.1S-1.5S

M19: Constructing Tests and Program Assessment

Knowledges 1.25K-1.31K Skills 1.24S-1.29S

M20: Multiple Choice Item Construction Knowledges 1.25K-1.31K Skills 1.24S-1.29S

M21: True False Item Construction Knowledges 1.25K-1.31K Skills 1.24S-1.29S

M22: Matching Item Construction Knowledges 1.25K-1.31K Skills 1.24S-1.29S

M23: Essay Item Construction Knowledges 1.25K-1.31K Skills 1.24S-1.29S

PPR Test Framework – Accelerate Online Module Alignment – Domain I

Domain II

PPR Test Framework Domain II: Creating a Positive, Productive Classroom Environment (approx. 15% of the test)

SBEC Standards Assessed PPR Standard II: The teacher creates a classroom environment of

respect and rapport that fosters a positive climate for learning, equity, and excellence.

M2: Time Management Knowledge 4.3K Skills 3.1S, 3.11S, 3.20S, 4.10S

M3: Discipline Management Techniques

Knowledges 2.6K- 2.23K Skills 2.6S - 2.21S

PPR Test Framework Domain III: Implementing Effective, Responsive Instruction and Assessment (approx. 31% of the test)

SBEC Standards Assessed PPR Standard I: The teacher designs instruction appropriate for all students that reflects an

understanding of relevant content and is based on continuous and appropriate assessment.

PPR Standard III:The teacher promotes student learning by providing responsive instruction that

makes use of effective communication techniques, instructional strategies that actively engage students in the learning process, and timely high-quality feedback.

Technology Applications Standards I-V

M1: Educational Philosophy – Approaches to Teaching

Knowledge 4.3K Skills 3.1S, 3.11S, 3.20S, 4.10S

M9: Diagnosing Learners for Instructional Delivery

Knowledge 3.5K - 3.11K Skills 3.7S - 3.14S

M10: Communication Skills Part I & II Knowledges 2.1K-2.5K 3.1K-3.4K 3.15K-3.16K Skills 2.1S-2.5S 3.1S-3.6S 3.18S-3.20S

M11: eCommunication Tools and Applications Technology Application – Knowledges: 2.1k, 2.3k ; Skills: 2.1s, 2.2s, 2.6s, 3.11s, 3.12s, 4.8s

M12: Basic Productivity Tools Technology Application – Knowledges: 1.3K; Skills: 1.3S, 3.1s, 3.2s, 4.03s, 4.04s

M13: Online Resource Applications Technology Application – Knowledges: 2.1k, 3.2k, 5.2k; Skills: 2.1s, 2.2s

M14: Inductive Instructional Delivery Strategies Knowledges 3.5K-3.11K Skills 3.7S-3.14S

M15: Deductive Instructional Delivery Strategies Knowledges 3.5K-3.11K Skills 3.7S-3.14S

M24: TrackStar I Technology Application –Skills: 2.01s, 2.02s, 2.06s

M25: TrackStar II (Project) Technology Application –Skills: 5.01s, 5.03s

M26: TrackStar III (Project) Technology Application –Skills: 4.08s, 5.03s, 5.07s

M27: Spreadsheets and Databases in the Classroom

Technology Application – Knowledges: 1.1k, 1.2k, 2.2k 3.1k; Skills: 1.5s, 3.2s, 3.3s, 3.15s, 4.2s, 4.9s

Domain IV

PPR Test Framework Domain IV: Fulfilling Professional Roles and Responsibilities (approx. 23% of the test)

SBEC Standards Assessed PPR Standard IV: The teacher fulfills professional roles and

responsibilities and adheres to legal and ethical requirements of the profession.

M16: Code of Ethics for Educators

Knowledge 4.1K-4.8K Skills 4.1S-4.11S

M17: Responsibilities to Educational Partners

Knowledge 4.1K-4.2K Skills 4.1S-4.4S

M18: Professional Growth and Continuing Education

Knowledge 4.9K- 4.12K Skills 4.12S - 4.15S

Data Collection

In general, each time a user moves to a new page, a page time stamp is recorded to indicate that page has been finished.

Page Time Stamp (PTS) –

Module Instructional Time

Module Instructional Time (MIT) – For a given module, every time a given user

accesses a module an estimate of the time spent on that module is calculated by computing the differences between each subsequent page time stamp (estimate of time spent on page). Then all times are added up for all pages associated with that module.

Data Sources – MIT Data Values being recorded in the eZone database: Time = Total MIT in seconds converted to minutes Time Info = Records MIT and total number of pages

visited Time Excluded = Very large PTS values are

excluded from MIT calculations. However, these values are recorded in the database as reference data.

Page Time Stamp (PTS) values 60 minutes are excluded.

(Rationale: Spending 60 minutes or more on one page is recorded as idle time)

Participants PPR Data

PPR Data for all Accelerate Online students was downloaded from the CEHD Data Portal and merged with eZone Data

PPR Total Score as well as the scores across the four domain areas were obtained

N=42 (Module data from 42 students were used)

N=28 (PPR data from 28 students were used)

Procedures

Calculated the total module instructional time and module averages for all students, across all modules

Data Analysis

Using the SPSS application program, correlation methods were employed to explore the relationship between PPR scores and module performance (completion time, posttest scores).

Findings - Preliminary

PPR Score by Total Module Instructional Time

R2 = .000 prob = .999

PPR Score by

Module Average

R2 = .525** prob = .006

Total Module Instructional Time and

Module Average

R2 = .115 prob = .420

Note: This investigation is still in progress

** Correlation is significant at the 0.01 level (2-tailed)

Discussion

The results from this preliminary investigation suggest that the amount of time spent on a module in an elearning environment is not related to the module posttest score or PPR score.

However, the posttest score of teaching candidates is significantly related to PPR scores

Limitations

Small sample size – I would like to complete this analysis in a few months, with a greater n

Initially I wanted to explore relationships between Module Clusters (1-4) and PPR Domains (1-4) – a more complete data set, and a larger sample are needed

THIS IS A WORK IN PROGRESS

Evolving Research Agenda

eZone –eLearning System

Online Modules ePortfolios Classroom Observations Integrated Resources

My eAssessment Center

Cluster 1

Cluster 2

Cluster 3

Cluster 4

Module/Course Assignments

Early Field Experiences

Instructional Artifacts

Domain 1

Domain 2

Domain 3

Domain 4

Domain 5

Community Discussion Boards

Community ePortfolios

Domain 6

Domain 7

Resource Media Center

Break Room

Phase 1

Phase 3

Phase 2

Evolving Research Agenda

How do various candidate performance data: Module Assessments Classroom Observations ePortfolio Assessment - Instructional Artifacts

Relate to key outcome measures (performance on PPR exams, program completion and retention, student performance …).

Results of Implementing Preservice Teacher Development Program

Year 1 Numerator Denominator Success Ratio

Candidates Placed/Total Candidates 10 10 1.00

Teachers Certified/ Total Candidates 8 10 .80

Teachers Retained/ Total Candidates 7 10 .70

Year 2

Candidates Placed/Total Candidates 19 20 .95

Teachers Certified/ Total Candidates 18 20 .90

Teachers Retained/ Total Candidates 17 20 .85

Year 3*

Candidates Placed/Total Candidates 40 46 .87

*The Year 3 Ratios of Teachers Certified and Teachers Retained are not provided because many of the Year 3 candidates are completing their final field experience.

Extras …

The results from this preliminary investigation suggest that engaged instructional time in an elearning environment within the context of a self-paced instructional module, doesn’t necessarily yield the same results as studies that focus on engaged time and academic learning time in face-to-face environments. Several studies involving f-2-f settings suggest that there is a positive relationship between time and student achievement. I would like to explore the literature more in these areas.

240.00

250.00

260.00

270.00

280.00

290.00

300.00

75.00 80.00 85.00 90.00 95.00 100.00

totalaver

Observed

Linear

Quadratic

PPR Score