Assessment in Language Learning

transcript

Assessment in Language Assessment in Language LearningLearning

Didi SukyadiDidi Sukyadi

Evaluation, Assessment ,and Evaluation, Assessment ,and Testing (Cameron, 2001:222)Testing (Cameron, 2001:222)

• Testing: One technique or method of assessment that is concerned with measuring learning through performance.

• Assessment: concerns with pupils learning or performance and thus provides one type of information that might be used in evaluation

• Evaluation: a process of systematically collecting information in order to make a judgment including the issues of lessons, programs through documentation, observation, interviews, questionnaires, etc.

The Place of Evaluation in The Place of Evaluation in Curriculum DevelopmentCurriculum Development

• Evaluation can or should be involved in all phases of curriculum development starting from needs analysis, stating the objectives, testing itself, material development, teaching and learning process and evaluation.

The Place of Assessment in The Place of Assessment in Teaching and Learning ProcessTeaching and Learning Process

• Brewster et.al. (2003:247): assessment plays an extremely important part in the teaching and learning process and may heavily influence the way the pupils are taught and the kinds of activities they do

Assessment in LearningAssessment in Learning

LEARNING = Teaching + Assessment

Outcome(Statement of competence)

(Statement of Achievement)

Learning(Alternative modes,

Contexts, Time-scales

Assessment(Alternative forms ofEvidence acceptable

TEST TYPESTEST TYPES

1) Selected response (binary choice, matching, and multiple-choice)

2) Constructed responses (fill in, short answer, performance format)

3) Personal responses (conference, portfolio, self assessment)

Constructed responseConstructed response

• Advantages: virtually no guessing factor, allows for productive language use, allows for testing the interactions of receptive and productive skills.

• Disadvantages: difficult and time consuming to score and subjective in scoring.

Selected responsesSelected responses

• Advantages: require a short time to administer, easy to score, scoring is objective.

• Disadvantages: relatively difficult to create, require no language production.

Personal Response ItemPersonal Response Item

• Advantages: directly related and integrated to curriculum, appropriate for assessing learning process.

• Disadvantages: difficult to create and structure, subjective in scoring

INTERPRETING THE OUTCOME INTERPRETING THE OUTCOME OF ASSESSMENTOF ASSESSMENT

1) Norm-referenced tests Any test that is primarily designed to

disperse performances of students in normal distribution based on their general abilities or proficiencies for purposes of categorizing the students into the levels or comparing students’ performances to the performances of others who formed the normative group (Glaser, 1963)

INTERPRETING THE OUTCOME INTERPRETING THE OUTCOME OF ASSESSMENTOF ASSESSMENT

2) Criterion-referenced tests

Measures which assess student achievement in terms of certain criterion standard thus provide information as to the degree of competence attained by a particular student which is independent of reference to the performance of others. They are deliberately constructed to yield measurements that are directly interpretable in terms of specified performance standard (Glaser and Nitko, 1971)

Other names for criterion-Other names for criterion-referenced testsreferenced tests

1) Domain-referenced tests (Documents that delineate a domain of student behaviors and the contents are materials to which test items are then referenced).

2) Objective-referenced tests (A test constructed so that the subsets of the items measures the specific objectives of a course, program of study or other clearly delineated subject matter area)

Characteristics of CRTCharacteristics of CRT

1) Emphasis on teaching/testing matches.

2) Focus on instructional sensitivity

3) Curricular relevance

4) Absence of normal distribution restrictions

5) No item discrimination restriction

CRT AND LANGUAGE THEORYCRT AND LANGUAGE THEORY

Two competing hypotheses

1) The divisibility of language ability

2) The communicative competence

The earlier rather simplistic views of language ability have been abandoned, recent focusing on performance assessment has raised new concern.

Nature of Language and Nature of Language and AssessmentAssessment

• Language and language acquisition are different in nature from other educational content such as in the relationship that exists in the nature of language proficiency and communicative competence.

• The difference will have a direct influence on how the construct of language knowledge is defined, how language tests are operationalised and how they are evaluated.

Language proficiencyLanguage proficiency

1) Functional approach (listing the various uses to which language can be put)

2) General proficiency (individuals differ basically in the measurable amounts of some indivisible body of competence they posses)

• CRT is very appropriate and useful in the assessment of such clearly definable but complex language tasks.

Communicative abilityCommunicative ability

1) Grammatical competence

2) Sociolinguistic competence

3) Strategic competence

4) Organizational competence

5) Pragmatic competence

A language test should reflect:A language test should reflect:

1) Language is used in interaction2) Interactions are usually unpredictable3) Language has a context4) Language is used for a purpose5) There is a need to examine a

performance6) Language is authentic7) Language success is behavioral based.

Testing communicative language Testing communicative language abilityability::

1) Be criterion-referenced against the operational performance of a set of language tasks.

2) Be concerned with validating itself against the criteria and be concerned with the content, construct and predictive validity.

3) Rely on modes of assessment which are qualitative

4) Subordinate reliability to face validity

Test itemTest item::

• A unit of measurement with a prompt and a prescriptive form for responding, which is intended to yield a response from an examinee from which performance in some language construct my be inferred in order to make some decision.

• A stem can be the portion of the item (in multiple choice), a quote the student must respond, or the reading passage that the student must analyze and write about.

WRITING TEST ITEMSWRITING TEST ITEMS

1) Do not explain too much.

2) Do not use trick questions

3) Provide only the information necessary

4) Avoid ambiguity

5) Be orderly in test presentation

Linguistic confoundingsLinguistic confoundings

1) Item should be written at the examinee’s level of proficiency

2) Item should not contain negatives or double negatives

3) Item should not be ambiguous• Family plays an important role in life. It

sometimes complicates matters. Explain this. Here this may refer to the role of the family or complication involved.

Format confoundingsFormat confoundings

1) Item should contain only relevant information.(1) Unnecessary information included• The following twenty vocabulary items have

been selected from the second reading texts in Unit 2 of the reading Packet. Your teacher discussed each of these words in class during the Wednesday vocabulary lesson ….

(2) Too brief• Write an essay comparing relationship in two

countries

2) Item should be independent

(1) What is the square root of 100?

(2) Multiply this by seven/

3) Item should be clearly organized and formatted.

• The item and its options should appear on the same page

VALIDITYVALIDITY

• Hughes (1989, 2003:26): a test is said to be valid if it measures accurately what it is intended to measure.

• Content validity: the content of a test constitutes a representative sample of the language skills, structures, etc.

• Criterion related validity: the degree to which results on the test agree with those provided by some independent and highly dependable assessment of the examinees’ ability,

• Construct validity: is the degree to which a test is measuring the psychological construct or constructs that it claims to be measuring

RELIABILITY (NRTs)RELIABILITY (NRTs)

1. Test retest reliability

2. Equivalent forms reliability

3. Internal consistency (split-half reliability

DEPENDABILITY IN CRTsDEPENDABILITY IN CRTs

• DEPENDABILITY IN CRTs1) Threshold loss agreement• Po = A + D N • Po = agreement coefficient• A = masters on both administration of the tests• D = non-masters on both administrations of the test• B = masters on the first administration but non masters

on the second.• C = Non masters on the first administration and master

on the second

ExampleExample

• Of the 45 examinees, 13 are categorized as A, 2 as B, 5 in C and 25 in D.

Po = A + D = 13 + 25 = 38 =

N 45 45

• Consistency due to the test itself

Po = (A + B)* (A + C) + (C + D)* (B + D)

Validating Test ItemsValidating Test Items

• Item Analysis

1) Index of difficulty/Item facility/Item easiness/P-value

2) Difference index

Item validityItem validity

1) Add item 1 to item 10 and you get the total score for each examinee (Score)

2) Item validity: Correlate each item with the score using point biserial correlation (correlating nominal and interval data)

Rater ConsistencyRater Consistency

1) Correlate score of each rater with the other two raters using Pearson product moment correlation.

2) If the correlation is significant, the rating is consistent.

AUTHENTIC ASSESSMENTAUTHENTIC ASSESSMENT

1) Real life, normal communication (the ability to perform particular tasks)

2) Interactional ability (total communicative effect)

What is meant by authentic?What is meant by authentic?

• Measures student’s knowledge and skills• Requires application of knowledge• Product or performance assessment• Relevant contextualized tasks• Process and products can both be measured• Part of learning process• Render holistic description• Reflection of real world

Types of Authentic AssessmentTypes of Authentic Assessment

• Oral interviews• Story retelling• Teacher observation• Experiments• Demonstration• Projects/Exhibition• Writing samples• Portfolios