1

TESL 4340 -- Evaluating CALL Packages:Curriculum/Pedagogical/Lingui

stics

Dr. Henry Tao GUO Email: henryguo@uic.edu.hkOffice: B 418

2

Outline

• Part one: Language program evaluation (Lynch 1996: 1-9)

• Part two: Evaluating CALL(Chapelle 2001:44-94)

3

Part one: Language program evaluation

• Critical issues

• Context-adaptive model (CAM)

4

Evaluation

• Evaluation here is defined as the systematic attempt to gather information in order to make judgments or decisions.

5

• As such, evaluative information can be gathered through different methods such as

ObservationQuestionaireInterview …

6

Why CALL evaluation?

• Internally, to better a program

• Externally, to justify program funding

7

8

Context Adaptive Model (CAM)

• Who needs the information? And what are the goals of evaluation? What are the roles of evaluators?

• Contextual questions• Which aspect of the program should the

evaluation focus on?• Data collection and analysis• Evaluation report

(Adapted from Lynch 1990a)

9

Who needs the information?

Audience

• Who is requesting the evaluation? • Who will be affected by the

evaluation?

Stakeholders (or clients)& Peripheral audience

10

What are the goals of evaluation?

• Why is the evaluation being conducted?

• What information is being requested and why?

The goals of evaluation may be determined by audience

11

What are the role of evaluator?

• The role of evaluator is determined by the particular audience and their goals

Someone providing consultation An expert standing in judgment A collaborator in program development A decision-making facilitator

12

• Hiring external evaluators for greater objectivity or internal evaluators for improving the curriculum?

13

Contextual questions

1. Is there a comparison group?2. Are there reliable and valid measures of

language skills?3. Are there various types of evaluation

expertise?4. Timing of the evaluation5. How students are admitted into the

program?

14

6. What the program students are like7. What the program staff are like8. Size and intensity of the program9. Instructional materials and resources10. Perspective and purpose of the program11. Social and political climate

15

Preliminary thematic framework (PTF)

• Where should the evaluator begin?

• What aspects of the program should evaluator investigate in detail?

16

An example of PTF developed for EST reading program

1. Effects of focusing instruction on reading only

2. Effects of focusing instruction on reading skills and strategies

3. Effects of using authentic reading texts

4. Feasibility of using students’ mother tongue versus target language for instruction

17

5. Availability of classrooms

6. Feasibility of using a ‘modified adjunct model’ approach

7. Feasibility and effects of conducting classroom-centered research

8. Levels of student proficiency in English upon entering the program

(Adapted from Lynch 1990a)

18

Data collection design/system

• What type of data need to be gathered-quantitative, qualitative or both?

• What will be the best method for gathering the data?

• The contextual questions can be used to determine which types of data collection design is possible.

19

Data collection and analysis

• How is data collection to be conducted?

• How is the results to be interpreted?

20

In the case of quantitative designs, have the assumptions of the design and statistical models been met?

In the case of qualitative design, have the procedures for data gathering been portrayed accurately, and have alternative interpretations of the data been pursued?

21

Evaluation report

• The evaluator must be sensitive to the audience and goals of the evaluation

• The social and political climate dimension of the context needs to be considered carefully at this stage

22

• The critical issue is how to communicate the findings of the evaluation honestly and successfully.

• Prepare multiple reports or express the findings in different ways, depending on the intended audiences.

23

Part two: Evaluating CALL

• The problem of instructed SLA

• Principles for CALL evaluation

• Judgmental evaluation of CALL

• Empirical evaluation of CALL

24

The problem of instructed SLA

• Cognitive conditions for SLA

• Social-affective conditions for SLA

• Other factors

25

Cognitive conditions for SLA

Choose a range of target structures Choose tasks which meet the utility

condition Select and sequence tasks to achieve

balanced goal development Maximize the chances of focus on form

through attentional manipulation Use cycles of accountability

26

Social-affective conditions for SLA

Willingness to communicate (WTC)

The desire to communicate with a particular person

Communicative self confidence at that particular moment

Interpersonal motivation Intergroup motivation

27

Self confidence Intergroup attitudes(e.g., integrativeness)Social situation(i.e.,features of context

affecting communication)Communicative competence Intergroup climatepersonality

28

Other factors

Individual differences in cognitive characteristics of learners due to, for example, age or cognitive style.

Learning situation such as the effects of task choice on teachers and learners and others who may be involved with the task

Practical factors such as the available resources

29

Principles for CALL evaluation

• Evaluation of CALL is a situation specific argument

• Judgemental anlaysis of software and planned tasks and empirical analysis of learners’ performances

30

• Criteria should come from theory and research on second language teaching

• Criteria should be applied in view of the purpose of the task

• Language learning potential should be the central criterion in evaluation of CALL

31

Judgmental and empirical analyses

32

Criteria from theory and research on SLA

• Language learning potential• Learner fit• Meaning focus• Authenticity• Positive impact• practicality

33

34

Criteria applied based on task purpose•These criteria for CALL appropriateness

need to be applied in view of the purpose of a CALL task.

•Moreover, tasks may have different purposes at various stages of instruction (Doughty & Williams,1998).

•Whatever the goal of the CALL task, however, evaluation of the task requires that it have a stated purpose.

35

The centrality of language learning

• Even though the importance of each of the six criteria may vary depending on the purpose of the tasks, language learning potential should be considered the most critical for CALL activities

36

Judgemental evaluation of CALL

37

38

Cases(Chapelle 58-67)

• Computer-assisted classroom discussion• Microworld• Text analysis• Storyboard• Concordancing

39

Summary

40

Empirical evaluation of CALL

• Why?

‘Students are often doing something very different from what [ language teachers ] assume they are doing’ ( Hosenfeld,1976:123). In other words, it is necessary to identify the observable data that provide evidence of CALL qualities.

41

How? Look for evidences

42

43

Language learning potential

• Focus on form• Modified interaction• Modified output

44

Learner fit

• Level of linguistic difficulty

• Individual difference

45

Meaning focus

• Effects of meaning-based instruction

• Assessing engagement with meaning

46

Authenticity

• Comparing CALL with non-CALL activities

47

Positive impact

48

Practicality

• Given the role of resources for the success of CALL, 1) some formal mechanism needs to be in place to monitor adequacy, and 2) an argument about CALL appropriateness should include a statement about sufficiency of resources.

49

Conclusion

• Evidence concerning the six ideal qualities of a particular CALL task needs to be combined to form an evaluative argument about the appropriateness of a CALL task for particular learners at a given point in time.

50

• Integrated research is needed to examine the types of CALL activates to seek evidence about CALL appropriateness in particular settings.

51

• From such research, implications can be drawn for specific contexts, and eventually research on particular types of activities might be generalised in a way that can inform general principles of CALL.