AMEP Assessment Task BankProfessional Development KitHow to Develop a Writing Assessment TaskNotes to accompany the PowerPoint for the AMEP Assessment Task Bank Professional Development Kit. Developed by Marian Hargreaves for NEAS, 2013.
Slide 1: Front Page
Slide 2: Writing is a good place to remind ourselves that assessment is an integral part of teaching and learning. “It helps to provide a picture of a … person's progress and achievements and to identify next steps in learning.” (Education Scotland, 2012)
Slide 3: (This slide has been enlarged and attached to the end of these notes for reference/as a handout)
Elizabeth Thomson’s “Effective Academic Writing” diagram, a variation of the teaching learning cycle, shows the interaction between key elements in building writing skills. Understanding these elements can assist teachers in planning and incorporating assessment as a logical/integral part of their teaching program. In a series of scaffolded developmental steps, learners gain control of a particular written text type and gain knowledge of how to use that text in social contexts.
The top left section ‘Building knowledge of field’ is a good place to start the process with group/class discussion of cultural context and shared experience. For example, when learners are preparing to write a formal letter, it would be helpful to look at different situations where letter writing skills are necessary or useful. This could take the form of a brainstorming activity, where learners reflect on the applicability of letter writing skills to their own settlement needs or pathway goals. The teacher can give input as necessary and introduce relevant vocabulary and grammatical patterns.
In the next section which involves modelling the text, learners focus on the register, staging and language features of the written text.
In the joint construction stage learners use the text type with support before moving to independent construction.
Note that the initial stages are all spoken text, and the process moves gradually to the written, more formal form, possibly after joint and independent construction has been done verbally.
While assessment of CSWE written texts generally takes place at the independent construction stage, the teacher can plan activities throughout the cycle which evaluate what the student already knows and determine what the learner needs next.
Slide 4: Purposes and methods of assessment
Ongoing assessment, as an integral part of the teaching/learning cycle, can have different purposes:
Placement (usually initial)
Diagnosis (for programming, identifying strengths and weaknesses for remediation)
Learning (to inform the process of learning and assessment for the students)
Progress (to monitor and inform both the teaching and learning process)
Achievement (to assess competence).
Ongoing assessment can use different methods
Progress checks and feedback by the teacher, the learners themselves (self-assessment) and by classmates (peer assessment), monitoring of class writing activities, writing in groups, pairs and alone are all examples of assessment methods that inform teaching and learning.
“Classroom investigations by AMEP teachers in writing classrooms support previous research evidence that self-assessment can provide learners with an ongoing learning tool as well as offering stimulus for authentic language interactions as they use and discuss its use with other learners and the teacher”. (AMEPRC, 2006)
Ongoing assessment can use written work done in different locations
You can assess in classroom time or accept a piece of writing done out-of- class. Apart from Distance Learning, writing assessments are generally done in-class for two principal reasons. Firstly, in most educational environments students have to write assessments under specific conditions eg time limits, so it’s good
Suggestion: Activity 1, a discussion on the prompts for a writing assessment task, could be done at this point.
practice, even though the CSWE does not have specified time limits. The second reason is that in-class writing demonstrates what the student can do by themselves, without outside assistance. When assessing writing done outside of class, it is assumed that the teacher will recognise whether the learners have in fact written the work themselves.
However, whatever the purpose, method or location, the principles of assessment still apply.
Slide 5: Cornerstones
Validity
Does the assessment task measure what you want it to measure? Does the content of the test represent the skills and knowledge that have been taught in the course? Do the assessment tasks and activities closely match the intended learning outcomes? Does the test enable learners to demonstrate the knowledge, skills and strategies that they have learned? Is it fair, or does its format/style/content favour some learners over others?
Reliability
Is the task consistent? Would it give the same result if repeated under similar conditions? Does it compare with similar tasks for similar students in comparable learning centres? Are the assessors who rate the written performances consistent from one student to another? Is there agreement between assessors on the same assessment task?
Rater moderation is therefore essential, so that teachers can be confident that they are marking to the same standard as their colleagues.
You can increase the reliability of writing assessment in a number of ways, for example through:
clear, unambiguous instructions and writing prompts having multiple raters participation in performance moderation and task validation.
Practicality
Assessments should be practical. They should not make unreasonable demands on learners or teachers. Your evidence gathering process should be manageable given the time and resources available, your mix of students, and their learning mode, for example distance learning. The assessment cycle should commence early enough in the teaching program to allow exposure to a range of model texts for a range of learning contexts.
Suggestion: Refer to the CSWE Conditions of Assessment as an important tool for ensuring reliability.
Any barriers to assessment should be identified, such as literacy lag, cultural, physical and psychological barriers, and taken into consideration.
Balance
There is always a balance to be struck between these three cardinal principles – sometimes the best possible assessment is just not practical. It is impossible to achieve 100% validity. When assessing the CSWE, using valid tasks and controlling the conditions of assessment as best you can will contribute to more reliable and valid assessment.
Integration of skills
Writing is a particularly good way to incorporate reading, speaking and listening skills, particularly if the teacher takes an integrated topic-based approach.
One size does not fit all students, and effective and appropriate assessment will recognise this. If possible, design and incorporate assessment from the perspective of the learner. This fits in well with helping the learner to understand and identify goals for their learning. A learner who wants to do a Certificate III in Aged Care will appreciate the need to be able to complete a form or write a report. Any student looking for a job will need to be able to write a resume and covering letter (CSWE III Module Q).
The choice of interesting and relevant topics is therefore very important. An interested learner will be better motivated, and hence more involved in their learning.
Slide 6: Task considerations
Barbara Kroll and Joy Reid identify 6 categories that test developers need to consider. These variables include:
1. Contextual. Both teachers and students need to know the context of the writing. When will the writing be expected to occur, where will it happen, how will it be scored, and what are the consequences?
2. Content. The topic and expectations embedded in the prompt must be within the experience of the students. Topics should be relevant and familiar. Interpretation of the topic may differ from that intended if a key word or phrase is misunderstood – a particular hazard with idioms. Cultural interference eg social norms and expectations can also lead to some unexpected and hence unsuccessful writing. Eg writing about a ‘blind date’ should not be interpreted as writing about someone who is blind.
3. Linguistic. Instructions to both the teacher and the student should be clear and unambiguous. These include the medium of writing (pen/pencil/computer), the paper to be written on (one side, or both? Special booklet or ordinary paper, lined?). How much time is allowed? A writing task should also not impose unrealistic reading demands on the learner who should be given the opportunity to clarify instructions before the test starts.
4. Task. Some tasks can sound deceptively simple. Eg ‘Describe your home’. But which home – the new one in Australia, the original one in the home country, the refugee camp where they spent the last several years?
The writing prompt can contain several tasks eg the stages of a narrative, making a choice and giving reasons for it.
5. Rhetorical. This relates to the way in which the writer is expected to deal with the content of the topic, and respond to specific instructions, such as compare and contrast. This is more relevant for academic writing than for AMEP students, but can still apply in cases such as a complaint to a landlord or retailer where a particular ‘voice’ is expected (ie firm but polite and quite formal).
6. Evaluation. Writing is often evaluated holistically, looking at the overall effect of the work. This can lead to some very subjective judgements and may result in poor rater reliability. The CSWE curriculum lists some very specific criteria which goes a long way to address this problem. However, both teachers and learners need to understand and remember the criteria. The prompts for writing should make sure that learners have every opportunity to meet the requirements of the criteria.
Slide 7: Components of a writing task
A writing task consists of:
The instructions for assessors – usually on a cover page. This will include guidelines for administering the task, and may also include a reminder of the conditions of assessment, for example, word length.
The rubric or instructions for learners. Information in the rubric should include the number of words the student needs to write, the timing (if any), and the procedures for responding. Rubrics should be kept simple for the student to understand, but sufficiently detailed for the student to know exactly what is expected of them.
The prompt. The prompt is the stimulus for writing and can also make clear the genre, purpose and audience of the task, for example, ‘write an email to your friend, telling them about your weekend’. Prompts should elicit appropriate writing that is representative of the learner’s skills and allow them to write to their potential.
Developing an assessment task begins with a clear understanding of objectives or the construct of the assessment.
Each CSWE module sets out the details of the criteria or objectives for each learning outcome.
Teaching programs need to incorporate these criteria or objectives so that students develop a critical awareness of them and internalise them as they practise and monitor their progress and achievement. Translated criteria are available to assist students to understand the criteria their performances are measured against for each CSWE outcome.
Ongoing integrated assessment enables both teachers and students to have a clear understanding of objectives and to take joint responsibility for progress and achievement.
Slide 9:
Step 2: Choice of topic
The choice of topic is the next major decision. Topics for writing assessments need to be familiar and relevant. They should arise from the content of the teaching program, which has been designed and developed to reflect student interest/need.
For example, if the topic is Transport, and the students have been developing skills and strategies to write recounts (CSWE II Module I Learning Outcome I2) there are a range of possible topics for assessment eg writing about an accident, a journey, buying a car, coming to Australia, going to a holiday destination.
Slide 10:
Step 3: Methods of assessment
There are a range of assessment methods described in the CSWE. Each module introduction lists suggested methods. One useful evidence gathering tool for assessing of writing skills is the portfolio. Generally, in the AMEP, portfolios consist of copies of written texts done by students over time to show the process of learning and improvement. Portfolios can be used together with self, peer and teacher feedback, eg regularly completed written texts can be assessed against familiar CSWE criteria, firstly by the writer, then a peer, and then a teacher with comments about what was done well and what could be improved next time.
Advantages of portfolios:
Support collaboration, development and revision Encourage student-centred teaching and learning Develop critical assessment skills Internalise task criteria
Encourage independent learning Judge ability from more than one or two pieces of work.
Disadvantages of portfolios:
Evaluation is time-consuming and labour intensive Judgements may be very subjective Require clear objectives and guidelines maintain inter-rater reliability and
scoring consistency Can disadvantage some students who have less access to resources,
especially if work is done outside the classroom.
‘Portability’ of writing
Can writing done for one task eg CSWE II I2 Write a recount, count towards a second learning outcome eg CSWE II J2 Write an informal text?
Yes it can, as long as the task enables the students to meet all the criteria for each learning outcome.
Slide 11: Student Writing and Plagiarism
Learners new to Australia often find plagiarism a difficult concept and one that may be actually contradictory to practices in their original country. If learners are encouraged to build their own learning and understanding and express their ideas and opinions verbally and in writing, they are less likely to borrow chunks of someone else’s work, and instead will be able to use other texts as models from which to develop their own writing style. If students are writing reports or the academic genres such as essays and discussions, it may be necessary to support students to extend their existing word power, use more complex grammatical structures and understand the differences between spoken and written language.
Slide 12:
Step 4: Piloting a new task
As with all other assessment tasks, a new writing task should be tried out with learners in a class. This will confirm if the criteria and conditions of assessment have been met, if the topic is a useful and relevant one, and if instructions to teachers and prompts to the learner are clear, unambiguous and produce the expected response. Badly written tasks not only fail to elicit good responses, but are unfair to students. Often a task that looks clear and straightforward to a native speaker is actually not at all clear and straightforward for the learners!
Slide 13: Activities – Activities for Writing Assessment
Activity 1: Assessment prompts – discussion
CSWE 2 (2008) Module K Learning Outcome 2 Write a recount
copies of the AMES sample tasks and the tasks on the Assessment Task Bank
Assessment criteria
Currently, both the AMES sample tasks and the tasks on the Assessment Task Bank present a very sparse assessment task. Learners are given lined paper with a topic specified, and the instructions that they must write approximately 100 words and may use their dictionary and/or spell-check.
It is assumed that the learner will have been taught, and can remember, all the defining features of a recount, can recall all the criteria that need to be addressed in order to achieve, and can generate the necessary scenario and sequence of events without assistance for this writing performance.
At least two samples of student writing should be used – one a benchmark piece, and another, borderline piece. Ideally, these should be from students at your learning centre, but if suitable pieces are not available, the AMES moderation resources are a good source of material for discussion, and also come with a Facilitator’s Guide:
Certificates I – III in Spoken and Written English, Moderation Kit (including separate booklet, Facilitator’s Guide), Moderation Resources, NSW AMES 2010, 2011, 2013.
1. Ensure that all participants are familiar with the criteria (and have a copy of the Assessment Marking Grid for the relevant Learning Outcome).
2. Discuss any particular criteria that your group considers problematic.3. Look at the benchmark piece of writing and discuss how it meets the criteria
of the LO.4. Look at the borderline piece of writing and discuss whether it meets the
criteria and, if not, where and why not.5. Aim to reach general agreement on the assessment and make notes for future
Activity 3: Develop an assessment for Cert III L1 Write a formal letter/email
Materials required:
Assessment criteria List of scenarios already available (see handout below)
1. Brainstorm a scenario. You may develop one already available (see handout below), or think of a completely different one.
2. Discuss and develop a structure that will assist the learner to plan a writing outline (criterion 1.1)
For example: what is included in a formal letter/email (date, address, subject/reference,
sign-off)(this should really be of a reminder of items already covered in class)
the points to be included in the complaint the suggested resolution possible timeframe for action, etc a reminder of the CSWE criteria for the learning outcome
Letter to a real estate agent (tap still dripping, despite recent visit from plumber)
Activity 4: Develop a writing assessment for a current unit of work/topic
Materials required:
CSWE curriculum The current unit of work or topic for which you want to design assessment
tasks
This activity is best done with at least two small groups, so that the assessment that is developed by one group can be swapped for comment with another group.
1. Identify the CSWE level for this activity and the Learning Outcome2. Look at the materials for the unit, or brainstorm the topic to identify teaching
and learning activities that would prepare learners for the assessment.3. Refer to the PPT notes from Slide 7: Writing Assessment Tasks, Step 1: clear
understanding of objectives.4. If brainstorming a topic, remember to make it
• relevant to the learners• interesting for the learners• appropriate for the current learning focus, for example work issues• practical to assess.
5. This activity is for a writing assessment but if you are brainstorming a topic, now is the time to think about how learning outcomes for the other macroskills could be integrated (and developed at another time).
6. Consider what type of assessment you want to incorporate into an integrated plan: formative/self or peer-assessment/summative for reporting on achievement.
7. Think about the evidence that you will require of the learner (eg portfolios, letters, emails, reports, descriptions etc) and the medium that will be accepted (handwritten, word-processed, poster).
8. Reference to the CSWE curriculum should be made at each stage of task development.
9. Design and develop appropriate and effective prompts for writing.10.Exchange your ideas/developed assessment with another group for peer
feedback.11.Pilot the final design of your assessment.
Slide 14: References
AMEPRC Fact Sheet. Teaching Strategies – 5: Using Learner Assessment for Writing Tasks.September 2006.
Elizabeth Thomson. Effective Academic Writing, Workshop. UNE 24 Nov 2012. Also available as a workbook: E Thomson and L Droga. Effective Academic Writing, PhoenixEducation 2012.
Education Scotland. Principles of assessment, inhttp://www.educationscotland.gov.uk/learningteachingandassessment (16 Nov 2012)
Alec Patton & Jeff Robin. Work that Matters Paul Hamlyn Foundation. February 2012
Sara Cushing Weigle. Teaching writing teachers about assessment. Journal of Second Language Writing 16 (2007) 194–209
B. Kroll & J. Reid. Guidelines for Designing Writing Prompts: Clarifications, Caveats, and Cautions. Journal of Second Language Writing, 3 (3), 231-255 (1994)
B. Song & B. August. Using portfolios to assess the writing of ESL students: a powerful alternative? Journal of Second Language Writing 11 (2002) 49-72
Venn, J. J. Assessing students with special needs (2nd ed.). Upper Saddle River, NJ: Merrill (2000). .
