For, With, By: Researchers and Teachers

CREOLE, City of Bristol College & CfBT Education Trust

Contextually Situated Research as a Driver Informing Policy, Practice & Theory

Professor Pauline Rea-Dickins

Overview of Presentation

Introduce the research: SPINE

Stakeholder involvement & partnerships

Research drivers, including the stakes &

consequences: real world problems

Some empirical findings from triangulated studies

Some conclusions

2

3 What is SPINE?

• Student Performance in National Examinations: the dynamics of language in school achievement (SPINE) www.bristol.ac.uk/spine (ESRC/DfID RES-167-25-0263)

• Bristol team: Pauline Rea-Dickins, Guoxing Yu, Oksana Afitska, Rosamund Sutherland, Federica Olivero, Sibel Erduran, Neil Ingram, Harvey Goldstein

• Zanzibar team: Zuleikha Khamis, Abdulla Mohammed, Amour Khamis, Mohammed Abeid, Shumbana Said, Haji Mwevura

4

Research Planning, Advisory Group Meetings

• Partnership

• Planning for the research

• Ministry of Education & Vocational Training SMT

Members – Advisory Group

• National Launch

• c. twice yearly updates of key findings

• Discussion of research communication

strategies/dissemination

• Channels for interaction, n.b. Znz Research Team

5

Why is MoEVT Interested? Supportive?

• Previous work by Bristol

• Institutional links and respect (e.g. doctoral training)

• Levels of achievement at end of Basic Education

alarmingly low

• Interest in SPINE: Evidence of policy changes English as MoI to be brought in earlier at primary (std4)

but commitment to raising children bilingually SUZA: universities do research

6

International Research Initiatives & Policy Decisions

Examinations in English: high stakes ‘real world’

problem Accommodations for ELL in US (e.g. Work of Abedi;

Bailey & Butler; Rivera et al) Teaching & learning of Science and Maths through

English: reversal of policy in Malaysia (see also Lan,

2010) Sri Lanka: at O and A level – students are using 2

languages to study & not compulsory to sit the exam in

EL2: they can choose (Punch, 2009)

7

Policy Issues: The Stakes & the Consequences in SSA

• Sub-Sahara Africa: what is the impact of an unfamiliar

language on learning progression & education outcomes?

• Research focus in sub-Saharan Africa more on classroom

interactions & subject learning with less attention for the

formal examining of this subject knowledge• Examining: Formal examinations at end of 2nd year of

secondary in English> 50% of school aged children leave school at the end of

Basic Education as unsuccessful learners: what does the evidence say?

8

Theory-focused Issues: Applied Linguistics

L2 academic language development across school

subjects

Working bilingually in the classroom

Monolingual (L1) examining

Negotiating double constructs in examinations

NS-orientation in examining; prescriptive vs supportive

assessment modalities

9

10 SPINE: theoretical issues - assessment

How valid:• is the dominant view of assessment as quality

measurement model vs. a supportive & context sensitive model (e.g. Pollitt & Ahmed 2009) + dynamic assessment research (Poehner 2008; Poehner & Rea-Dickins 2010)

• is the role of a dominant world language in LTA processes

• are current constructs, e.g. that assume NS norms, and monolingual performance as the normal context for language use

11 SPINE: EXEMPLAR FINDINGS (STUDY 5.1)

What are the four things that man could resemble

whales?

45 students took this item:– 35.6% = no answer– 26.7% = wrong answer– 28.9% = partially correct answer– 8.8% = correct answer

12Int: D1 – No response to Q3 explains (STUDY 5.1)

D1: “because I did not understand by this … this … resemble” (lines 115-117)

Int: “If I tell you that resemble means ‘to look like’ … can you do the

question now?D1: “Yes”Int: OK so what’s the answer?D1: “Man … is warm blooded … and

whales also … whales have lungs and man also have lungs …” (122-133)

Original Items: Summary of Findings (STUDY 5.1)

Very low mean scores across subjects

Student achievement lowest in Maths

Significant differences in achievement across the subjects

Strong correlation between English (exam + vocabulary

test) & performance in other subjects

Factors affecting performance from learner interviews: Not understanding the task, specific words & phrases or the

meaning of tables and diagrams Partial knowledge of topic area & question type Low levels of students’ language proficiency

13

Modified Items - some findings (STUDY 5.1)

• Increase in student response rates

• Increase in accuracy of responses

• Correspondence between ability to provide a correct

translation of task & ability to solve/provide partially

correct answer

• Learners who had difficulty translating task generally

performed poorly

Evidence of linguistic factor + other factors

14

COURSEWORK ASSESSMENT : implementation issues (STUDY 4)

Significant variability among teachers:

• the types of specific activities that constitute teacher

assessment

• the number of activities from which they take marks for

each month

• decisions on how to handle student absenteeism

• how to arrive at the CWA mark: some picked the one

that students did better on, some picked randomly, and

some averaged of all marks

16Schools A-F – limited exposure to EnglishTests of significance: comparison of students performance across the 3 different versions and per subject (STUDY 5.3)

ANOVA

4087.505 2 2043.753 43.666 .000

17411.053 372 46.804

21498.558 374

125.327 2 62.663 1.274 .281

18246.171 371 49.181

18371.497 373

12.587 2 6.293 .256 .774

9149.103 372 24.594

9161.689 374

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

biology

Chemistry

maths

Sum of Squares df Mean Square F Sig.

17

language version

EnglishBilingualKiswahili

Mea

n of

bio

logy

20

18

16

14

12

10

8

18 Qualitative Analysis of Student Responses(STUDY 5.3) Top performing students

Qn 5: Write about an animal you have studied

• Name of the animal:

• What does the animal look like?

• Where does the animal live?

• Describe how the animal eats?

Qualitative Analysis of Student Responses(STUDY 5.3) Top performing students

Reaching potential: analysis of higher performing

students (see handout)

19

20

Findings from National Form II Data: summary of the multilevel modelling analysis(STUDY 2)

• It is very clear that ENGLISH language proficiency is a significant and substantial predictor of the students’ performance in MATH, BIO & CHEM.

• But: the school-level variances explained in the cons models as well as in the models including ENGLISH as the single explanatory variable demonstrated that a substantial proportion of the variance is attributable to school factors

21 CONSEQUENCES: High stakes classroom assessment - the realities & impact

Impact/Potential Disadvantage

Consequences/Injustice

•Teachers do not use full range of LTA procedures & processes•Use of inappropriately constructed assessment frameworks•Inaccurate CWA of learners•CWA implemented as a series of tests

•Learners not fully supported in their language & content knowledge development & fail to reach potential •Test performance valued over learning •Leave school with poor educational outcomes•Inadequate d-base for decision making about student & learning progression

22 Enhancing Learning & Social Justice

Impact/Potential Disadvantage (examples)

Consequences/Injustice: (examples)

•Learners do not engage or respond poorly in examinations•Subject area (e.g. Biology, maths) construct can only be assessed where a linguistic construct has been successfully negotiated

•Loss of self-esteem & motivation for learning•Learners fail to reach their potential (glass ceiling effect) or fail altogether•Leave school as unsuccessful (e.g. at end of Basic Education)•Unequal access to available resources, educational experiences & work opportunities •Unskilled & unable to join the workforce in turn contributing to social & economic deprivation

Research Evidence as Driver for Change? For Awareness Raising? (1)

For policy:

• Potential through partnership: SUZA, MoEVT

• Triangulated evidence, progressively focused studies

• All learners affected, not only lower achieving learners

For professional practice

• Stakeholder engagement with findings e.g. through

workshops

For community

• Through media, e.g. Kiswahili press

23

Research Evidence as Driver for Change? For Awareness Raising? (2)

For Language Testing and Assessment Community/International Examining Boards

• What evidence is there of the impact of research findings such these on actual testing and assessment policies and practices? (e.g. Shohamy)

• Time to overthrow the dominant & unquestioned role that EL1 has in many examining contexts: time for a coup!

• Time to reconceptualise constructs: NS orientations• Investigate supportive & context sensitive approaches

to assessment

24

Research Evidence as Driver for Change? For Awareness Raising? (3)

• Where is the most impact likely?– For policy?– For professional practice?– For community?– For LTA/Applied Linguistics community– For International Examining boards?

• Where do partnership strengths lie?

25