notes on medical education assessment

KMTC Dept. of Medical Education

Assessment Assessment modulemodule

Norbert Boruett

A Pragmatic approachA Pragmatic approach

With Norbert Boruett

assessment units

Units in the assessment module

Unit one – principles of assessmentUnit two- concepts associated with

assessmentUnit three- assessments instrumentsUnit four -Integratory

assessment units

Assessment- try to visualise Assessment- try to visualise

why should we assess? what should we assess ? How should we assess and when should we assess?

assessment units

Effects of assessment on learning and teaching Assessment influences what is taught and

what is learned. Often teaching is artificially divide from assessment. The two teaching and learning are linked by feedback(Harden,1992).

. We shall discuss more on feedback on subsequent parts of this module

assessment units

Effects of assessment

Torrance (1993) asserts that changing theories and methods of assessment have been the focus of significant attention for some years now. Curriculum developers have realised that real change will not take place in schools if traditional paper and pencil tests, be they essay or multiple choice, remain unchanged to exert a constraining influence on how teachers and pupils approach curricula.

assessment units

Aschroft and Foreman- Peck (1994) point out that assessment is a key element in the effective management of learning than any component of the learning process.

Curriculum developers have noted that positive change will not take place in learning institutions if assessment does not change (Tornance,1993).

Effects of assessment on learning and teaching- cont

assessment units

Miller (1976) puts it more succinctly by asserting that to change the curricular without changing examination will achieve nothing.

He further observes that changing the examination systems without changing the curriculum had a more profound impact upon the nature of learning than changing curriculum without altering the examination.

Effects of assessment on learning and teaching- cont

assessment units

Effects of assessment on learning and teaching- contThe bicycle is a useful model when

considering assessment and teaching. Teaching and learning are represented by front wheel, assessment by the rear wheel. Disasters may occur when the two wheels go in different directions. If the rear wheel is omitted altogether only one wheel remains: difficulties and problems arise. In the same way problems may arise if a curriculum is designed with no assessment component

assessment units

Assessment and evaluation-?.

Aschrof and Foreman – Peck (2000) highlight that assessment is often confused with evaluation. Assessment can be conceptualised to mean judgment about the progress (formative assessment) or achievement (summative assessment) of a particular course of study or a particular teaching intervention (Nayar, 1995).

assessment units

Formative assessment

There is a problem with formative assessment since continuous assessment is frequently used summatively.

Breaks learning into manageable unitsAllows repeated attempts to master the

content of each moduleIs not perceived as threatening (low stakes)

assessment units

Summative assessment

Is the end point examinationCan block intended career

progression(high stakes)Is perceived as threatening

assessment units

Criterion and normative assessmentAssessment can be seen as criterion or

normative based. Brown (1994) postulates that criterion based assessment tells us what has been achieved. He further articulates that this type of assessment should have a description telling us what has not been achieved and the reason behind the failure. Put, succinctly criterion referencing is designed to check what a student has mastered in an area of knowledge or skill.

assessment units

Tools in assessment and evaluationBrown (1995) asserts that the same tools

may be used to assess and evaluate at the same time, for example it is possible to use one tool to evaluate a teaching method and assess a student. If a student does well in an examination, it follows then that the teaching method was appropriate. This example implies the function as an evaluation of the teaching received.

assessment units

Conclusion

As a principal objective of medical education is to produce a competent physician , then , unquestionably , the raison d'être for the evaluation process is to assess the competency and not the rank order of students. To this end, criterion – reference testing is necessary and must become the principal method of evaluation within medical education( Turnball,1989)

assessment units

Who designs an administers the assessment ?Traditionally the teacher has been

instrumental in executing this task. Perhaps you may now be aware that

currently there exist assessment methods where students take sole responsibility and much-pronounced role in their assessment

Examples here include portfolio-learning, self-assessment, peer assessment (Brown, 1992; Boud, 1995).

assessment units

Assessing ability- competence

Hubbard adage states that:

There is something more scarce,

Something finer far,

Something rarer than ability.

It is the ability to recognise ability

assessment units

OSCE-as an answer

Lumley (etal., 1999) assert that OSCE brings a new dimension to assessment of medical training. Of particular importance is the ability to examine practical and other skills in a unified, measurable and reproductive fashion.

assessment units

OSCE CONT

They are effective in assessing knowledge and practical skills and ensure that each student is presented with the same material, thus providing a uniform evaluation with the consistent marking of all those involved.

Do you recall the Kenyan experience in OSCE?

assessment units

Why OSCE’S?

Construct validity is the ability of the OSCE to differentiate students ability , or follow a student’s progress before and after a course of instruction.

Content (criterion- based) validity assesses the value of the station in reaching its specified objective.

assessment units

Why should students be Why should students be assessed? assessed?

Safeguarding the public Feedback to the students Certification Monitoring the programme

assessment units

Feedback

Rowntree has laid down a claim that feedback, or knowledge of results, is the lifeblood of learning.

Feedback is valuable – hence it should not consist of notification of pass or failure. Feedback should be given immediately and comprehensive at the end of an examination (Dunnington et al., 1992). Feedback should be given positively (Amiel and Lunenfeld, 1992).

assessment units

Guidelines for giving feedback

Timing of feedback· . Methods of providing feedback

Students checked their answers against the correct ones written on the board. Students discussed the questions and answers with an instructor. Students were assigned to textbook materials relating to questions they had answered incorrectly

assessment units

Feedback methods cont

Give student’s checklist with examiners ratings

Watching video tape of own performance Discussion with examiner Watching video tape of expert Receiving feedback from patient or

standardised patientReading computer generated report of

performance

assessment units

What should we assess?

Knowledge (Cognitive) Skills (Psychomotor) Attitudes (Affective

assessment units

How should we assess?

Validity- ensure the examination is valid Reliability- ensure that the exam is reliable Feasibility- set a realistic exam that can

measure what you want to.

assessment units

When should students be assessed?

During the course of training

At the end of the course

assessment units

Formative assessment

The scoring of formative assessment should be strictly criterion referenced so as to indicate to students their level of performance. Aschrof and Foreman – Peck (2000) asserts that formative assessment should support student learning , records of achievement , learning conflicts, portfolio record of competence.

assessment units

quip

Examinations are formidable even to the best prepared , for the even the greatest fool may ask more than the wisest man can answer.

Charles Caleb Colton ,1780-1832

assessment units

Concepts associated with assessment Concepts associated with assessment

Validity- measuring what you purport (want)to measure

Reliability- consistency.

assessment units

Types of validity

Face validity-Content validity Predictive validity Consequential validity Construct validity

assessment units

Type of validity Test facet being measured

Questions asked

Face validity Compatibility with educational philosophy

What is the test face value? Does it match up with educational intentions?

Content validity The content of curriculum Does the test include a representative sample of the subject matter?

Construct validity The ability to differentiate between groups with known difference in ability(beginners versus experts

Does the differentiate at the level of ability expected of candidates at the stage in training?

assessment units

Type of validity Test facet being measured

Questions asked

Predictive validity The ability to predict an outcome in the future eg professional success after graduation

Does the test predict future performance and level of competence?

Consequential validity The educational consequence of the test

Does the test produce the desired educational outcome?

assessment units

Reliability

Inter-rater reliability- which correlates the consistency of rating of performance across different examiners

Inter-case reliability which quantifies the consistency of performance of candidate across the cases

assessment units

Reliability

Parallel form reliability Test retest reliability

assessment units

Estimating reliability

Split halfCronbach alpha coefficient

how can we handle reliability issues?

assessment units

Norm referencing and criterion

what is the difference ?

assessment units

Why use norm referencing tests?

As you answer that - think of the inherent dangers of norm referencing

Outline some of them

assessment units

Criterion-referenced tests

How is this different from the former? And what are the advantages and disadvantages of this referencing visa –vis the former

assessment units

What are the practical issues od assessment designUtility=

reliabilityXvalidityXfeasbilityXacceptabilityXeducational impact- Cees van der Vleuten 1996

assessment units

References

1.Talpot M. Monkey see and monkey do: a critique of the competency model in graduate medical education. Med Edu 2004:38 587-92

2.Wass V,Vleuten CPM van der,Shatzer J,Jones R. Assessment of clinical comptence.Lancet 2001: 357 945-9

assessment units

Assessment instruments

assessment units

Types classroom assessment

Placement assessment• Screening assessment• Formative assessment• Summative assessment(Berk R A, 2002)

Are the students qualified, to advance to the next, course level?

assessment units

Millers Pyramid

Types of classroom assessmentPlacement assessment• Screening assessment• Formative assessment• Summative assessment(Berk R A, 2002)

assessment units

Exercise

Reflect for a moment and write downassessment instruments that youknow and how they influence yourlearning!Remember difference between method and

instrument:Method of travel = by land, sea, air…Instrument of travel = train, bus, ship,

plane…

assessment units

Taxonomies

Classification of objectives or skills• Hierarchic– To achieve higher level one mustmaster lower one• Motivation for holistic form ofeducation

assessment units

Bloom’s taxonomy

assessment units

Millers pyramid

assessment units

Millers pyramid

assessment units

Millers pyramid

assessment units

Validity

Proper assessment instrument?– Thermometer -> Temperature– Thermometer x-> Weight– MCQ -> Know & understand

– MCQ x-> Show, demonstrate, do

assessment units

• Content (=direct) validity– Relevance?– Representative?– Subjective!

• Construct (=indirect) validity– Test result -> Competence?

assessment units

Reliability

Test-retest reliability• Equivalent forms reliability• Split-half reliability• Scorer/rater reliability

– Inter- vs. intrarater reliability

assessment units

Factors in clinical skills assessment that lead to lower reliability

Two few stations or too little testing timeCheck list that do not discriminate (too

easy or too hard)Unreliable patients or inconsistent role

playersIdiosyncratic examinersAdministration problems

assessment units

Engaging with data

Write down some observationsabout the data. What do you see?

2. Suggest explanations (hypotheses)for the data.3. How would you test the hypotheses?

assessment units

Feasibility

Cost in terms of resources and time of Construction

• Marking• Organisation• Quality feedback

assessment units

Utility

• U = Utility• R = Reliability• V = Validity• E = Educational impact• A = Acceptability• C = Cost

assessment units

Types of assessment

Multiple Choice Questions (MCQs) Modified Essay Questions (MEQs) Patient Management Problems

(PMPs) Traditional essays questions Structured questions .filling the blanks True or false questions

assessment units

Able to do

assessment units

assessment units

12 curriculum outcomes. SMS

What the

Clinician is able

to do-doing the right

thing

How the clinician approaches

Their practice-doing the thing

rightThe clinician as

professional

The right

person doing it

assessment units

What the doctor is able to do 1-7

Competence in clinical skillsCompetence to perform proceduresCompetence to investigate a patientCompetence to manage a patient Competence in health promotion and disease

controlCompetence in skills of communicationCompetence to retrieve and handle information

assessment units

How the doctor approaches their practice 8-10With understanding of basic and clinical

sciences With appropriate attitudes, ethical

understanding and understanding of legal responsibilities.

With appropriate decision making skills and clinical reasoning and judgment

assessment units

The doctor as a professional 11-12

Appreciation of the role of the doctor within the health service

Attitude for personal development

assessment units

Multiple Choice Questions

They are designed to test knowledgeOutcomes assessed- knowledge of basic

skills and clinical sciences8patient investigation3 Patient management 4 health promotionClinical reasoning ands judgment

assessment units

Multiple Choice Questions

They are designed to test knowledge

Outcomes assessed- knowledge of basic skills and clinical sciences8patient investigation3 Patient management 4 health promotionClinical reasoning ands judgment

assessment units

What do multiple choices questions measure Knowledge of specific facts Knowledge of terminology Knowledge of principles Knowledge of procedures The understanding and application of

facts and principles The understanding of proceduresReasoning to some extent

assessment units

How to construct MCQs

State the question positively Construct options that are grammatically

consistent with the item. Do not use MCQs when other instruments

are more appropriate.

assessment units

Multiple choice questions- principles(rules) The question itself or the stem must be

appropriate Have one correct response to the

question Distracters or the incorrect responses

among the choices should be plausible

assessment units

Extended Matched Items

Management of diabetes in the community?

1.

1. start on oral hypoglycemic2. do diabetic annual review3. three monthly blood tests and review4. monthly blood tests and review5. Refer to podiatrist6. refer to hospital diabetes specialist nurse7. Trial diet control for 3 months8. 2start on twice daily insulin injection 9. Do fasting blood sugars10. Do haemoglobin A/c blood test

assessment units

Extended matching

Select the most appropriate course of action for the following patients

1. Woman, mid 20s ,14 weeks pregnant to have glycosuria on routine resting in GP ante natal clinic

2. Woman,mid 50s ,BMI 30, just found to have fasting blood sugar 15mmol/l,HBA a18.5%

3. South Asian woman, mid 30s complaining of fatique, thirst and dry mouth

assessment units

Extended Matched Items (EMIs)

Purpose – test knowledge and reasoning skills

Outcomes assessed- all outcomes with an emphasis on knowledge and clinical science8patient investigation3 Patient management 4 health promotion5Clinical reasoning ands judgment

assessment units

MCQ

Multiple Choice Questions are used to tests students’ factual knowledge and to some extent the critical thinking of students. An attempt is made to test higher levels of Bloom’s taxonomy and not mere recall of facts.

Integratory

assessment units

1. A cervical smear taken from a thirty five year old lady was found to have the following:

Mitotic cells Multinucleated cells Loss of chromatin granules Cells that have lost adhesiveness

assessment units

Mcq

Which of the following will be the best classification of this smear?

a) CIN Ib) CIN IIIc) CIN IVd) CIN Ve) CIN II

assessment units

Which of the following will be the best classification of this smear?

a) CIN Ib) CIN IIIc) CIN IVd) CIN Ve) CIN II

assessment units

2. Tissue Processing in Histology is the most critical stage in entire processes that go on in a Histology lab. Which of the following will determine the processing procedure?

Urgent diagnosis The element to be demonstrated The materials availableThe skills involvedThe size of tissue for examination

assessment units

MEQ

Modified Essay Questions will be used to assess the students’ ability to integrate facts and problem solving skills, which are fundamental to good practice. Question requiring consolidation of facts and critical thinking will be asked.

assessment units

1. A tissue section was found to be having a brown colour that was intracellular upon staining with H/E staining technique. The pigment was found to be soluble in alcoholic picric acid.

assessment units

Possible scenarios

i. What is the possible name of this pigment?

ii. What features enabled you to identify the pigment?

iii. What class would this pigment fall into?Assuming the pigment was not soluble in

alcoholic acid picric acid, how would this affect your identification

assessment units

Formatively as learning tool to stimulate

discussion and top plan future development

Can be used to stimulate and provide feedback – may be every three months what is going , intriguing / exciting ,worrying phenomenon.

assessment units

What can MEQ measure ? MEQ can be used as a formal assessment

technique It can be used as an aid to learning. It can be used as an adjunct to a lecture. It presents good questions for group

discussion. It can be used as a self-instruction method.It can be used as a self-audit device.

assessment units

Construction of MEQ

Drawing up a draft of the subject area. Circulating the draft to a team of fellow

professionals to study it. Each of the professional colleagues should be

asked to provide the answer to each of the questions.

These responses should be studied and forms the basis for model answers.

The marks should be allocated according to consensus decisions

assessment units

PATIENT MANAGEMENT PROBLEM (PMP)

Patient management problem (PMP) consists of short questions that require also short answers that revolve around managing a patient. They are intended to equip the student with knowledge, skills and attitude that are necessary in professional practice.

The main feature is that a series of questions are asked about a case and options are provided that leads to proper management of the patient. It tests students on a wide range of subjects.

assessment units

What PMP can assess

Like the MEQ, PMP provides the opportunity for students to demonstrate their ability in patient management skills and problem solving skills in a variety of scenario.

The students are presented with simulated patients, real patient or imaginary patient in a paper and pencil format. The unfolding scenario is presented logically and the student is evaluated on the basis of the proper steps taken leading to the proper management of the patient.

assessment units

Constructing a PMP

Constructing a PMP begins with a description of the symptoms of the patient and the resources that are available. The student is presented with the problem followed by a series of questions relating to management, diagnosis and investigations. Each question has a number of options from which the student must select. Beside each option are hidden instructions about how the student should proceed.

assessment units

Steps in constructing PMP

Clearly define what it is you wish to test in the problem. Set realistic objectives for the group you are testing.

Select a problem that is meaningful and suitable. Ensure that the sequence of management is logical. Prepare the opening scene. This should compose

of: the actual setting for the case, the presenting features of the patient, and the role of the user, the resources available and clear instructions.

assessment units

Steps continue

Prepare a list of clinical choices which are consistent with the problem.

Pretest the PMP using a group of subject experts. Also try it out with one or two potential candidates.

Score the PMP. Allocate a mark for each individual choice on the PMP.


Work-based Assessment

Assessing learning acquired in the workplace

assessment units

Work-based Learning

Learning acquired from and through work experience

Usually involves a learning contract that outlines what the person will undertake to achieve from the work experience

Also involves a mentor from the health service in the workplace

Involves academic staff in assessing and accrediting work-based learning

assessment units

Work-based Learning

Brings together educational providers and service providers and increases relevance

Particularly useful for continuing professional development (CPD)

Economically attractive since the person remains in the workforce providing service

Requires academic staff to develop new competencies in relation to supervision, assessment, and distance learning

assessment units

The use of Work-based Learning in higher education can facilitate the development of situational, practical, process and people-oriented knowledge (Eraut, 1994)

It can encourage on-going reflection on knowledge, values and practice, thus increasing the professional’s self-awareness and understanding of the social factors that affect performance, such as health service policy, management and organisation.

assessment units

WBL requirements

The award to which the learning contributes must be within a credit accumulation and transfer system (CATS)

Credit must be based on achievement of specific learning outcomes - preferably competencies

Assessment procedures and instruments have to be performance-based

assessment units

Assessing competencies

Assessment that draws on evidence of professional competence:

tests a wider range of knowledge, skills and attitudes and, also, how well they are integrated in practice

encompasses a balance of propositional and process knowledge through performance

assessment units


feeds back to teaching and learning and helps workplace mentoring by assessing against competence criteria

allows flexibility and participation in assessment, with students judging the right time to be assessed

prioritises assessment that is conducted in real work settings, providing authenticity and feedback to the learner

assessment units


helps to make transparent the state of the art in professional practice (through analysis of competencies) and assists in its on-going improvement

facilitates the setting of reliable and valid national standards, and the marketing of marketable and transferable awards

links in with other approaches that stimulate reflection on education and practice (e.g. formative assessment, profiling and records of achievement, self-assessment)

assessment units

The framework for approaches to WBL assessment

General competence criteria set in an integrated, holistic framework provide the basis for assessment in professional contexts

Competence is a construct inferred from the available evidence of skills, knowledge, attitudes and values

Authentic assessment tasks present open-ended situations that test professional judgement

assessment units


Routine performance and reproductive thinking give way to higher-order, creative thinking that:is complexyields multiple solutionsrequires application of multiple criteriaoften involves uncertaintyfosters self-regulation in the thinking processseeks to create meaning in experience

assessment units


Need for a multi-dimensional model that reflects the interplay in medical practice of professional attributes that are knowledge-based and experienced-based scientific knowledge, judgement, patient management skills,

technical skills, interpersonal skills, attitudes and personal characteristics, experience monitoring and effectiveness of health interventions

WBL assessment has to measure the practitioner’s success in synthesising from this repertoire

assessment units

Approaches to WBL assessmentWill the assessment be focused on Process

or Outcome?Will the assessment provide

Hard (quantitative) or Soft (qualitative)measures?

assessment units

Hard - Process assessment

Frequency counts of conversations, body language, requests for learning support

Time on taskAuditMonitoring of computer data

assessment units

Hard - Outcomes assessment

Multiple assessors - managers, colleagues, patients

Performance checklists/ Rating schedulesObservation/video of skills demonstration

or simulated tasksMultiple choice tests/ Modified essaysWritten report (structured)Oral presentation/poster

assessment units

Soft - Outcomes assessment

Observation of learner attributes, motivation, inquiry orientation

Generic skills/ transferProfilesPersonal/ joint reportsSelf- or peer assessment

assessment units

Soft - Process assessment

Summative statementObservation of body language, learner

interactions, consultations with tutor/supervisor

Dialogue around learning strategies, log books/diaries

Cognitive mapping, reflexive self-learning

assessment units

Portfolios

Best way of achieving a holistic pictureMulti-dimensional and integrate evidence

gleaned from a range of methodsBring together the results of performance

ratings, supervisors’ judgments, and assessed work such as project reports

Present evidence of personal reflection and critical evaluation

assessment units

Assessing Group Work

This method compliments efforts of small group teaching. As task is given to students and marks awarded. They further clarify that the task may be divided to give individual marks. There are a number of disadvantages in this type of assess, including:

- Dysfunctional group dynamics - Cultural factors - Members who don’t value small group

teaching.

assessment units

Diaries

Students’ records can be kept as a supplementary or additional form of assessment. Ashcroft and Foreman-Peck (2000) assert that it can be set as an alternative to more traditional forms of assessed writing.

assessment units

Self-Assessment

Self-assessment is being used more an more in higher education as a strategy for both student learning and assessment (Boud, 1977). It is a key skill to be developed on any course. Self-assessment is not a new techniques, but a way of increasing the role of students as active participants in their own learning (Boud, 1995).

assessment units

Self assessment continued

All assessment – whether conducted by teachers or by learners – involves two key elements.

· The development of knowledge and appreciation of the appropriate standards and criteria for meeting those

standards that may be applied to any given work. Unless it is known

what counts as good work, it is impossible to tell whether the specific work being considered is

adequate. · The second is the capacity to make judgements about whether or not the work involved does or does not meet

those standards.

assessment units

Peer Assessment

This is the assessment involving students. Students give themselves qualitative feedback to fellow colleagues during a presentation. Feedback may also be given on written work. Giving feedback is a tricky affair, students may become defensive. Students may possibly be asked not to reject comments. Assessment should be made on agreed criteria made by the students.

assessment units

Learning Contracts

Learning contracts may be seen as agreement between a student and a tutor about what and how a student should be learning. Ashcroft and Foreman (2000) suggest a well designed pro-forma, detailing out what objectives are to be agreed on. The contracts can be re-negotiated.

assessment units

Records of Achievement

Records of achievement sometimes call profiles, are systematic attempts to involve students in recording, reviewing and evaluating their academic and personal development through out the college career.

assessment units

Portfolio assessment

A portfolio is literally ‘a collection of papers ’Portfolio based learning is a method of encouraging reflective learning for professions. A professional development portfolio is a collection of materials, made by a professional, that records, and reflects on, events and process in that professional’s career (Hall, 1992).

assessment units

Records of Achievement

Records of achievement sometimes call profiles, are systematic attempts to involve students in recording, reviewing and evaluating their academic and personal development through out the college career.

assessment units

Portfolio assessment

A portfolio is literally ‘a collection of papers ’Portfolio based learning is a method of encouraging reflective learning for professions. A professional development portfolio is a collection of materials, made by a professional, that records, and reflects on, events and process in that professional’s career (Hall, 1992).

assessment units

What do portfolios offer?

They recognize and encourage the autonomous and reflective learning that is an integral part of professional education and development

They are based in the real expereince of the learner, and so help the connection between theory and practice.

They cater for a range of learning

assessment units

What do portfolios offer?cont

They enable assessment within the framework of clear criteria and learning objectives.

They can accommodate evidence from different sources.

They provide a process for both formative and summative assessment , based on either personally derived or externally set learning objectives

assessment units


They provide a model for life long learning and continuing professional development

assessment units

What is the educational rationale

promote adult and learner cantered learning -Portfolios provide educational opportunities that are interactive and reflective and can be related to real life experience.

In many ways ,portfolios may be considered the ultimate educational tool in terms of meeting the criteria for good practice in adult learning.

assessment units

Portfolio learning characteristics

Learning impacts on expertise in several ways

Making changes in practice Seeking answers to new questionsValidating established practices

assessment units

What does a portfolio look like?

There is no common for a portfolios range from simple notebooks and diaries to larger A4 folders and computer held files. Some learners prefer to given something , and A4 szed file with a dairy inserting has been welcomed by many . However being prescriptive can generate resistance, and flexibility over format is important .

assessment units

What can be included in portfolio ?Critical incidents or events with patientsA reflective journal or dairyTutorials and learning plans and reflection

on themExam preparation materialVideo recording of consultations and other

relevant material

assessment units

What can be included in portfolio ?Audits and project workCritical reviews of articlesFeedback materialManagement material

assessment units

Critical incidents of events with patientsCritical incidences are those incidents in a

working day that are memorable for going well or badly, or which have proved generally thought provoking(snadden et al.1999).

assessment units

Reflective journal or diary

Putting thought into paper this encourages the learner to reflect into what they are doing.

assessment units

Tutorials and learning plans

Descriptions of tutorial that have happened and the follow up of learning needs that have resulted from them can be used.

The areas of strengths and developments are noted and future references can be made to see if learning has taken place

assessment units

How can the material in a portfolio be used?As a method of promoting personal

developmentFormatively as a learning tool to stimulate

discussion and to plan future learningAs a formal (Summative) assessment tool.

assessment units

As a method of personal development and way of tracking progressPortfolios can be used for self learning

activity without support and input from a tutor or mentor.

Learners collect material to track down their own progress

Network of tutors and mentors would make the learning more sucessful

assessment units




assessment units

Assessing portfolios

Assessing portfolios presents a number of particular challenges because of the highly individual nature of the portfolio.

The process does not fit in the tradition concept of assessment in medical education.which has its roots in the science of objectivity.

In portfolio we must have room for subjectivity and artistic appreciation of the work presented

assessment units

Portfolios assessment may have the following aims:

To provide feedback to learners so that can learn from mistakes and build on achievements

To motivate learners and focus their sense of achievements

To enable learners to correct errors and remedy deficiencies

To consolidate learning

assessment units


To help learners to apply abstracts principles to practical contexts.

To guide selection,option or career choice To classify or grade learner achievementsTo estimate potential to progress to other

levels or coursesTo give teachers feedback on how effective

they are at promoting learning

assessment units


To provide statistics for internal and external agencies

assessment units


To give teachers feedback oh how effective they are promoting learning

To provide statistics for internal and external agencies(Brown and Knight, 1994).

assessment units

Strengths of portfolios

Portfolios have their strengths in providing a method of giving feedback, in helping learners define their strengths and weakness and in providing evidence of achievement towards set learning objectives. These objectives can be set by the learner or by an external body(Snadden et al.,1999).

assessment units

As a formal (summative ) assessment tool

Assess practice over a period of time- it assess performance in practice over a period of time. Authentic assessment.

Useful in formal and summative assessment in nursing(Jasper,1995).

Snadden and Thomas ,1996: 1998point out the negative effect of using portfolio as an assessment tool.

assessment units


assessment units

Critical incidents of events with patientsCritical incidences are those incidents in a

working day that are memorable for going well or badly, or which have proved generally thought provoking(snadden et al.1999).

assessment units

Reflective journal or diary

Putting thought into paper this encourages the learner to reflect into what they are doing.

assessment units

As a method of personal development and way of tracking progressPortfolios can be used for self learning

activity without support and input from a tutor or mentor.

Learners collect material to track down their own progress

Network of tutors and mentors would make the learning more sucessful

assessment units




assessment units





assessment units



To provide statistics for internal and external agencies.

assessment units

Strengths of portfolios

Portfolios have their strengths in providing a method of giving feedback, in helping learners define their strengths and weakness and in providing evidence of achievement towards set learning objectives. These objectives can be set by the learner or by an external body(Snadden et al.,1999).

assessment units

As a formal (summative ) assessment tool

Assess practice over a period of time- it assess performance in practice over a period of time. Authentic assessment.

Useful in formal and summative assessment in nursing(Jasper,1995).

Snadden and Thomas ,1996: 1998point out the negative effect of using portfolio as an assessment tool.


O S C EO S C E

assessment units

What is an OSCEOSCE ??OObjective SStructured CClinical EExamination

ObjectiveObjective – means fair and without bias. Most examination in the world are not fair. Use of checklist ensures objectivity.What is your comment?

StructureStructuredd refer to the organization of the examination. Why use OSCE . The rationale.

assessment units

O.S.C.E. cont.

Globally many patients are gaining rights and becoming fewer. Thus students may not gain meaningful clinical experience.

During examination periods students get to be examined in different and sometimes cases that do not pose the same difficulty. What would be the effect?

assessment units

This type of examination is characterized by several several stationsstations, that reflect aspects or areas a clinician as to master. Think of some roles a doctor/ nurse must be able to do.

ClinicalClinical – the all examination entails the clinical aspects of a an health worker. In other words the station are clinical in nature.

ExaminationExamination – this summarizes the all episode. It is an examination with usually declares those who are safe to handle the public. Larger market share

O.S.C.E. cont.

assessment units

Goal and Objective of an Goal and Objective of an

OSCEOSCE Produce competent health workers Help in the learning process of the

students and a humanistic assessment method

Allow for meaning research and staff development.

assessment units

WHY OSCE

Validity issues – most examinations do not measure What they purport to measure. Think of many theory examination that we really on

Reliability – consistency many examination set ups are not consistence in the way it determines failures and those who have performed well

assessment units

Typical observer sheet- rating scale/ check list must have the: Name of std,skill, tasks

Competence Yes No

Greets patient

Explains procedure

Assembles equipment

Withdraws medicine and expels air

Swaps the area the correct area

Inserts needle horizontally and gently propels medicine


Observer sheets- Weighting the skills

A mark is always provide against any correct maneuver. However some skills must be weighted heavily because of their importance

assessment units

Scoring the skill

Add all the correct procedures Double the score as and when indicatedDivide the total score obtained against the

total possible scoreScore= correct score

Total possible score

assessment units

OSCE

Requires a lot of tedious planning

Requires skilled personnel

Requires substantial resources

Demands a lot of dedication on the part of the participants


Integratory

unit

assessment units

Integratory

This integral unit covers various assessment instruments and an elaborate procedure of developing Objective Structured Clinical Examination (OSCE) is also discussed.

Integratory

assessment units

Assessment and learning outcomes

Many medical schools are developing learning outcomes or goals that expect their graduates to achieve at the course often termed exist outcomes.

They be expressed as competencies, roles, outcomes or goals or knowledge, skills and attitudes

assessment units

Assessment and learning outcomes- contThe wording of learning objective give glues to

the type of assessment instrument to be used.Recall ,name, define, state, describe, be aware, understand’

refer to knowledge and therefore are generally assessed by knowledge based examination such as MCQ, extended matched items.

explain recognize ,review, discuss, identify, distinguish, debate, analyse, assess design, refer to the interpretation of knowledge and ability to reason- this are assessed by EMI,s, structured responses questions essay and project work

assessment units

Assessment and learning outcomes- contThe wording of learning objective give

glues to the type of assessment instrument to be used.

Demonstrate ,use ,apply, perform, show, be able to, refer to skills that you need to acquire that may be assessed in OSCEs, long cases or OSLERs.

assessment units





assessment units



To provide statistics for internal and external agencies.

assessment units

The The OSCE ProgrammeOSCE Programme

assessment units

The The OSCEOSCE Programme Programme

What is an OSCE ? Norbert BoruettNorbert Boruett

assessment units

OSCEAn examination

gaining global appeal

assessment units

What is an OSCE ?Objective Structured Clinical Examination

Objective – means fair and without bias. Most examination in the world are not fair. Use of checklist ensures objectivity.What is your comment?

Structured refer to the organization of the examination. Why use OSCE?. The rationale.

assessment units

Clinical – the all examination entails the clinical aspects of an health worker. In other words the station are clinical in nature.

Examination – this summarizes the all episode. It is an examination with usually declares those who are competent to handle patients.

O.S.C.E. cont.

assessment units

Goal and Objective of an OSCE Help in the learning

process of students Produce competent

health workers Allow for meaning

research and staff development.

assessment units

Think?

WHY OSCE ?

Validity issues – most examinations do not measure WHAT they purport to measure.

Reliability – consistency many examination set ups are not consistence in the way it determines failures and those who have performed well

assessment units

Station

This is the region where the skill is demonstrated by the candidate

The assessors check () against the competence shown in check list.

i.e. giving an injection

assessment units

Station RotationStation Rotation

Station 1Station 1 Station 2Station 2

Station 3Station 3

Station 4Station 4

Station 5Station 5Station 6Station 6

Station 7Station 7

ININ

OUTOUT

assessment units


The Observer Sheet Norbert BoruettNorbert Boruett

assessment units

Typical observer sheet: Name,skill, tasks

Competence Yes No

1. Greets patient

2. Explains procedure

3. Assembles equipment

4. Withdraws Medicine and expels air

5. Swaps the correct area

6. Inserts needle horizontally and gently

7. Propels medicine

assessment units

Observer Sheets Weighting the skills

A mark is always provided against any correct maneuver. However some skills must be weighted heavily because of their importance!

assessment units

Scoring the skill

Add all the correct procedures Double or triple the score as and when

indicated Divide the total score obtained against the

total possible score Score=correct score / Total possible

score

assessment units


How to Design an OSCE?

assessment units

Design and Run an OSCE?!

Teamwork!

assessment units


Teamwork!

Brainwork!

assessment units


Teamwork!Brainwork!

Think work!

assessment units


Teamwork!Brainwork!Think work!

Venture!

assessment units


Teamwork!Brainwork!Think work!Venture!

Tour-de-Force!

assessment units


BUT… It is a real Challenge!It is absolutely Worthwhile

assessment units

BASIC CRITERIAFor the Ideal Skills Assessment

To be Well Founded and Sound-Based To have Validity and Reliability

To have a Practical Manageability

The Assessment has:The Assessment has:

assessment units

BASIC CRITERIUM 3Practical Manageability The number of Students to be assessed? The available Staff Members The available Workspace,

accommodation & requisites To what extend the obtained assessment

results can be used in an objective way To let pass or let fail the student To filter out the poor or the bad student

assessment units

Step 1Step 1

The Scenario for the OrganisationThe Scenario for the Organisation

Of Typical OSCE SessionOf Typical OSCE Session

Step 2Step 2

Step 3Step 3

MonthsMonths

WeekWeek

DayDay

Preparatory TasksPreparatory Tasks

The Week- & Day-The Week- & Day-Before TasksBefore Tasks

The Day-Itself TasksThe Day-Itself Tasks

assessment units

Step 1Step 1 Preparatory TasksPreparatory Tasks

Fix dates of the OSCEFix dates of the OSCE day(s) day(s)

Staffing Staffing Observers Observers

Test SubjectsTest Subjects

Students to pass the OSCE Students to pass the OSCE

Accommodation facilitiesAccommodation facilities Rooms Rooms

ManikinsManikins

AttributesAttributes

assessment units

Step 1Step 1 Preparatory TasksPreparatory Tasks

Identify the StationsIdentify the Stations Number of stations Number of stations

Skill in each stationSkill in each station

Design Observer SheetDesign Observer Sheet

Consent on gradingConsent on grading

Secretarial TasksSecretarial Tasks Announcements Announcements

SubscriptionSubscription

Control systemsControl systems

Roster designsRoster designs

Task distributionTask distribution

assessment units

Step Step 22 Week & Day Before TasksWeek & Day Before Tasks

Are All Students InvolvedAre All Students Involved properly Informed?properly Informed?

Suspension of all Classes, lecturers, practicals etc?Suspension of all Classes, lecturers, practicals etc?

Is Administration and Direction fully informed ?Is Administration and Direction fully informed ?

Are All OSCE Staff Member committed to their Are All OSCE Staff Member committed to their several tasks?several tasks?

assessment units

Step Step 22 Week & Day Before TasksWeek & Day Before Tasks

Organise the SET of the OSCEOrganise the SET of the OSCE

Station Rooms, Requisites, Manikins, Station Rooms, Requisites, Manikins, Waiting RoomsWaiting Rooms

Security of the Content of the StationsSecurity of the Content of the Stations

Final Rehearsal en BriefingFinal Rehearsal en Briefing

Programme in minute detail is distributedProgramme in minute detail is distributed

assessment units

Step Step 33 The Day Itself TasksThe Day Itself Tasks

GENERAL RULEGENERAL RULE

An OSCE Runs on the StopwatchAn OSCE Runs on the Stopwatch

All students present 15 minutes before All students present 15 minutes before launchlaunch

Date post:	05-Dec-2014
Category:	Documents
Upload:	norbert-boruett
View:	416 times
Download:	2 times

notes on medical education assessment

Documents