MEASURING QUALITY IN INITIAL IN TEACHER EDUCATION (MQUITE)

Aileen Kennedy, University of Edinburghon behalf of the MQuITE team

20th September 2018@DrAileenK #MQuITEwww.scde.ac.uk

THE STORY OF MQUITE…

Scottish Government commissioned longitudinal study

(2017 – 2022)

Need for sector to be

proactive

Lack of appropriate

ways of assessing the quality of ITE

A changing context

MEASURING QUALITY IN INITIAL TEACHER EDUCATION (MQUITE)

• Funded by Scottish Government and supported by GTCS, partners include the Universities of:• Aberdeen, Dundee, Edinburgh, Glasgow, Highlands and

Islands, Royal Conservatoire, Stirling, Strathclyde and West of Scotland (to include QMU & Napier)

• Aim of project: • to develop, collaboratively, a framework for assessing ITE

quality and to use it as part of a five-year cohort study• Research questions:

1. How can quality in ITE be measured in a Scottish, context appropriate way?

2. What does this measuring tell us about aspects of quality in different ITE routes in Scotland?

THEORETICAL FRAME

• Developing a ‘contextually-appropriate framework’ is an explicit act of ‘vernacular globalisation’ (Appadurai, 1996)

• or, as Rizvi & Lingard describe: ‘the ways in which local sites and their histories, cultures, politics and pedagogies mediate to greater or lesser extents the effects of top-down globalization’ (2010, p. 65)

MQUITE AS PART OF A BIGGER RESEARCH AGENDA FOR TEACHER

EDUCATION

‘The problem [...] is that the weight of the research, being fragmented, often narrowly focused, and usually not directly connected to a shared research agenda on teacher education, does not position teacher educators strongly to craft an evidence-based narrative about teacher education that might counter policies and reports like the NCTQ’s [National Council on Teacher Quality, Washington D.C. think tank].’

(Sleeter, 2014, p.152)

SLEETER (2014) CALLS FOR:

1. Collaborative studies of ITE that bring together a number of researchers

2. Studies that are embedded and connected to a wider, shared research agenda in teacher education and that will contribute to building a sound knowledge base for the field

3. Studies which produce useful evidence and conclusions of interest to policy-makers responsible for shaping education

THE LITERATURE REVIEW

Scope and methodology: • conceptual review; • publications in English; • published since 2005.

1. Contexts and trends in the field of ITE2. Approaches to conceptualizing and measuring ITE

quality3. ITE quality frameworks and studies

WHAT DID THE LITERATURE REVIEW REVEAL? (1)

Most individual studies of ITE quality either:• Correlate individual student attainment data

with the teacher’s ITE programme (Kirabo Jackson et al., 2014); or

• Review programme documentation, teachers’ academic qualifications and absence records (Jacob & Welsh, 2011)

Approaches tend to be either:• Educational-sociological • Educational-psychological

WHAT DID THE LITERATURE REVIEW REVEAL? (2)

Three conceptualisations of quality frameworks:

• Shared values (e.g. Boston College Social Justice Model)

• Professional standards (e.g. Stanford STEP model)• Theoretical model (e.g. Australian SETE spatial-

based research design)

WHAT DID THE LITERATURE REVIEW REVEAL? (3)

Gaps in the literature:

1. Partnerships2. Teacher educators

KEY MESSAGES IN DESIGNING A QUALITY EVALUATION FRAMEWORK

• There is no one ‘proven’ way to measure quality in ITE• ‘Quality’ is a context-driven concept• Any evaluation framework requires a clear and shared

articulation of its purpose• More than one type of evidence is necessary (e.g.

surveys, documentary analysis, observation), as is assessment of more than one aspect (e.g. programme structures, teacher educator quality, student teacher performance)

IMPLICATIONS FOR MQUITE

• Meso-level study is a potentially very valuable contribution to the field

• Importance of a longitudinal study

• Clarity around key aspects of our context and purpose of the evaluation project

• Importance of: • a learning and development orientation • voice for the sector

THE PROCESS OF DEVELOPING OUR FRAMEWORKTeam members

read lit. review and highlighted what

was relevant/important

Shared these views with whole

team

Agreed on Feueret al's categories as starting point

Contextualised categories: 'Scotified'

Draft version shared more

widely

Final version of agreed

categories

COMMONLY ASSESSED COMPONENTS OF ITE QUALITY

(FEUER ET AL., 2013)

1. Admissions and recruitment criteria2. Quality and substance of instruction3. Quality of student teaching experience4. Faculty qualifications5. Effectiveness in preparing new teachers who are

employable in the field6. Success in preparing high-quality teachers

1. ADMISSIONS AND RECRUITMENT CRITERIA

• Associated with selectivity, reflects quality and/or rigour?

• High academic achievement on entry = ‘high quality’ recruits?

• Teacher effectiveness influenced by other factors too, e.g. institutional location, prestige, status of profession locally, labour market needs etc. (Labree, 2010)

• ‘High quality’ recruits often leave the profession in higher numbers

2. QUALITY AND SUBSTANCE OF INSTRUCTION

• Frequently measured through analysis of programme syllabi

• Can identify inconsistencies and gaps in course work and integration with fieldwork

• Assignments and course materials can also be analysed, but require consistently-applied criteria

• BUT such measures may not reflect what is actually taught, or its quality and effectiveness

3. QUALITY OF STUDENT TEACHING EXPERIENCE• Fieldwork policies

• Reveal programme design, but not necessarily student experience or its impact

• Qualifications of placement school mentors• Doesn’t identify the quality of the mentoring

• Observations of students teaching• Requires shared rubric for assessment, and consistent

application• Surveys of teacher candidates

• Perception studies; perception may change over time• BUT these are short-term, immediate measurements

4. FACULTY QUALIFICATIONS• Demographic information re. percentage of Faculty:

with higher degrees; full or part time; permanent or temporary

• Teacher educator quality less frequently assessed (Snoek et al., 2010)• Neglect of this aspect is ironic, given the global

consensus that ‘teachers matter’, and by extension, therefore, ‘teacher educators matter’ (Goodwin, et al., 2014)

5. EMPLOYABILITY OF NEW TEACHERS

• Commonly measured by: pass rates on licensure and data on hiring and retention• Sometimes linked to programme duration and type

(Ingersoll et al., 2012)

• Tatto et al. (2016) claim programmes most likely to show positive employment outcomes have:• Resources to invest in longitudinal efforts• Strong social/institutional networks to support field

experience• Guaranteed employment at end of programme

6. SUCCESS IN PREPARING HIGH-QUALITY TEACHERS

• Teacher performance assessments at end of programme and/or end 1st year teaching

• Graduate/employer surveys• BUT Tatto et al (2016) warn of the assumption that

higher levels of satisfaction necessarily indicate higher levels of quality

• Mixed evidence on the accuracy of principals’ assessments of teacher effectiveness (Coggshall et al, 2012; Jacob & Lefgren, 2008)

• Value-added approaches to measuring impact on pupil learning: number of limitations

Feuer et al’s (2013) categories

MQuITE considerations

1. Admissions and recruitment criteria

Admissions, recruitment and retention

2. Quality and substance of instruction

Assumes instruction (teaching) takes place only in the university context.

3. Quality of student teaching experience

Conceptually problematic positioning of teacher mentors and of school-based ‘instruction’

4. Faculty qualifications More complex than this: mix of staff, experience, preparedness. Raises questions about who ‘teacher educators’ are.

5. Effectiveness in preparing new teachers who are employable in the field

Uptake of ‘Teacher Induction Scheme’

6. Success in preparing high-quality teachers

Post-registration destinations, longitudinal

RELEVANT ASPECTS NOT PRESENT IN FEUER ET AL’S CATEGORISATION

Partnership Institutional context

MQUITE FRAMEWORK COMPONENTS

Components Relatedspecific dimensions

Data collection tool

1. Partnership2. Admissions, recruitment and retention3. Programme design4. Practicum/fieldwork5. Teacher educators6. Initial destinations7. Post-registration8. Institutional context

DATA COLLECTION TOOLSA. Programme profileB. End of ITE surveysC. Annual GTCS data checkD. End of induction surveyE. Annual 2018 cohort surveyF. 2018 cohort individual interviewsG. Induction manager focus groupsH. ??

FIRST PHASE OF EMPIRICAL DATA COLLECTION

• Three surveys live in May/June/July:• 2018 graduates (323 responses)• School staff involved in mentoring/supporting

students (229 responses)• University staff involved in ITE (150 responses)

WE ASKED ABOUT…

• Preparedness, competence and confidence• Partnership• Selection, development & delivery of ITE• Site-based and university-based learning• Assessment and the SPR

HEADLINE MESSAGES FROM STUDENT SURVEY

• Confident to begin teaching, but apprehensive

• Good levels of confidence in teaching core subjects and BGE

• High degree of satisfaction with university-based learning

• Highly valued site-based experiences

HEADLINES MESSAGES FROM SCHOOL STAFF SURVEY

• Lots of experience of mentoring students

• High feelings of competence• Some mentor

training/education• Large appetite for more

partnership working• In the main felt supported by

university• Large proportion feel that site-

based learning is effective• Desire to contribute more to

development and delivery of ITE

HEADLINES MESSAGES FROM UNIVERSITY STAFF

DATA• Some degree of specific training

for their role• Large numbers participate in

selection procedures• Perceived need for greater

partnership• Much agreement that university-

based learning is useful• Much agreement that site-based

learning is important• High confidence in students’

preparedness as beginning teachers

SOME AREAS TO EXPLORE MORE DEEPLY

• How do different stakeholders perceive ‘quality’ in relation to beginning teachers

• Nature of partnership: what is it, whose responsibility, how can it be enacted?

• Qualifications/training of staff: what is appropriate, how might this take place?

• Nature of practice in school/university: what might this be, what might the distinctions be?

APPLYING THE MQUITEFRAMEWORK IN OTHER NATIONAL

CONTEXTS

Affordances• Explicit

acknowledgement of process of ‘vernacular globalisation’

• Possibility of building cumulative knowledge

Constraints• ‘Ready made’

framework by-passes the collaborative development phase

• Contextualisation aspect may prove to limit possibility of cumulative knowledge

CONTACT ME…

Dr Aileen Kennedy, University of Edinburghaileen.kennedy@ed.ac.uk

@DrAileenK #MQuITE

www.scde.ac.uk

REFERENCES• Appadurai, A. (1996). Modernity at large: Cultural dimensions

of globalization. Minnesota: University of Minnesota Press.• Feuer, M. J. et al. (2013) Evaluation of Teacher Preparation

Programs: Purposes, Methods, and Policy Options. Washington, D.C.: National Academy of Education.

• Jacob, B.A. & Welsh, E. (2011). What’s in a rating? Economics of Education Review, 30, 434-448.

• Kirabo Jackson, C., Rockoff, J.E. & Staiger, D.O. (2014). Teacher effects and teacher-related policies. Annual Review of Economics, 6, 801-825.

• Rivzi, F. & Lingard, B. (2010). Globalizing education policy. Abingdon, Oxon.: Routledge.

• Sleeter, C. (2014) ‘Toward Teacher Education Research That Informs Policy’, Educational Researcher, 43(3), pp. 146–153. doi: 10.3102/0013189X14528752.

REFERENCES• Blömeke, S. et al. (2008) ‘Effectiveness of teacher education’, ZDM, 40(5),

pp. 719–734. doi: 10.1007/s11858-008-0096-x.• Coggshall, J. G., Bivona, L. and Reschly, D. J. (2012) Evaluating the

Effectiveness of Teacher Preparation Programs for Support and Accountability. Research & Policy Brief. Washington, D.C.: National Comprehensive Center for Teacher Quality. Available at: https://eric.ed.gov/?id=ED543773 (Accessed: 5 May 2017).

• DeAngelis, K. J., Wall, A. F. and Che, J. (2013) ‘The Impact of Preservice Preparation and Early Career Support on Novice Teachers’ Career Intentions and Decisions’, Journal of Teacher Education, 64(4), pp. 338–355. doi: 10.1177/0022487113488945.

• Feuer, M. J. et al. (2013) Evaluation of Teacher Preparation Programs: Purposes, Methods, and Policy Options. Washington, D.C.: National Academy of Education. Available at: https://eric.ed.gov/?id=ED565694 (Accessed: 10 May 2017).

• Goodwin, A. L. et al. (2014) ‘What Should Teacher Educators Know and Be Able to Do? Perspectives From Practicing Teacher Educators’, Journal of Teacher Education, 65(4), pp. 284–302. doi: 10.1177/0022487114535266.

• Ingersoll, R., Merrill, L. and May, H. (2012) ‘Retaining Teachers: How Preparation Matters’, Educational Leadership, 69(8), pp. 30–34.

REFERENCES CONT’D

• Jacob, B. A. and Lefgren, L. (2008) ‘Can Principals Identify Effective Teachers? Evidence on Subjective Performance Evaluation in Education’, Journal of Labor Economics, 26(1), pp. 101–136. doi: 10.1086/522974.

• Labaree, D. (2010) ‘Teach for America and Teacher Ed: Heads They Win, Tails We Lose’, Journal of teacher Education, 61(1–2), p. 48.

• Sleeter, C. (2014) ‘Toward Teacher Education Research That Informs Policy’, Educational Researcher, 43(3), pp. 146–153. doi: 10.3102/0013189X14528752.

• Snoek, M., Swennen, A. and van der Klink, M. (2010) ‘The Teacher Educator: A Neglected Factor in the Contemporary Debate on Teacher Education’, in Hudson, B., Zgaga, P., and Åstrand, B. (eds) Advancing Quality Cultures for Teacher Education in Europe –Tensions and Opportunities. Umeå, Sweden: Umeå School of Education, Umeå University, pp. 33–48. Available at: http://idiprints.knjiznica.idi.hr/196/1/2010%20Domovi%C4%87,%20Vizek%20Vidovi%C4%87.pdf#page=13 (Accessed: 7 June 2017).

• Tatto, M. T. et al. (2016) ‘The Emergence of High-Stakes Accountability Policies in Teacher Preparation: An Examination of the U.S. Department of Education’s Proposed Regulations’, education policy analysis archives, 24(0), p. 21.