Date post: | 07-Feb-2017 |
Category: |
Education |
Upload: | liz-norman |
View: | 55 times |
Download: | 0 times |
Oral exams
Liz NormanMassey University
insight reliability
Oral exams: strengths and limitations
insight reliability
Oral exams: strengths and limitations
• depth of knowledge• analytical and reasoning
capabilities• cognitive processes• decision making abilities• oral communication
insight reliability
Oral exams: strengths and limitations
• depth of knowledge• analytical and reasoning
capabilities• cognitive processes• decision making abilities• oral communication
• few cases/topics sampled• lack of examiner
agreement• assessing candidates on
different cases/topics• assessing different
numbers of cases/topics
insight reliability
Oral exams: strengths and limitations
• Ability to tailor questions and probing increases insight
• Long cases allow you to explore depth
• Standardisation of questions and probing increases reliability
• More short cases increases reliability
insight reliability
Oral exams: strengths and limitations
• Ability to tailor questions and probing increases insight
• Pre-prepared probes and marking schemes increases reliability
insight reliability
Oral exams: strengths and limitations
1. Finding the right balance2. Reducing examiner-examiner unreliability
Reducing examiner-examiner unreliability
• Examiner training– Use of exemplars of different standard– Practice in assessment
• Advance preparation of questions and prompts– Well-structured questions aimed at assessing appropriate skills– Clear and straightforward to answer
• Advance preparation of marking schemes– Common picture of performance levels: Familiarisation,
discussion, and agreement between examiners on the marking schemes
– Well structured marking schemes which are easy to apply and well-understood by examiners
Reducing examiner-examiner unreliability
• Allowing only preplanned questions and prompts– Planning for contingencies
• Putting candidates at ease as much as possible• Reducing examiner load as much as possible• Monitor examiner agreement• Consensus decision making
Finding the right balance: blueprinting
• Don’t aim to assess factual knowledge in oral exams – written exams are better for this
• Aim to assess:– Analytical, problem solving, reasoning skills– Cognitive processes– Decision-making abilities– Judgement– Oral communication– Professionalism, values, ethics (if these are part of
the subject guidelines)
Finding the right balance: questions
• Make your questions short, straightforward and clear
• Centre on clinical scenarios• Get straight to the point.
– Remove unnecessary information– Consider even saying what the point is:“This question is going to be about (disease x) and I want to explore your approach to management.”
• Keep Qs short - aim for new Q every 5 mins
Finding the right balance: questions
• Make Qs open-ended – Require interpretation, problem solving,
reasoning, judgement, or decisions– do not have one right answer
• Do not ask for personal experience or opinion– Eg: “How is this usually managed in your
practice?”, “What is your opinion on?”– Its “unmarkable” content– Can throw candidates. Differential effect on those
not “testwise”
Broad types of oral Qs
Recall: Questions that ask candidates only to recall facts, principles, or other information.
– List the most common causes of (x).– Define (x).– What is the most common disease associated
with (x)?– Explain two mechanisms by which (x) results
in (y).– Describe the differences between Type 1 and
Type 2 (x).
McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.
Broad types of oral Qs
Recall: Questions that ask candidates only to recall facts, principles, or other information.
Interpretive: Questions that require interpretation of information, and using principles to deal with new situations.
– Describe the changes on this radiograph.– Interpret the results of this biochemistry panel.– Describe the abnormalities you see (in this image).
McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.
Broad types of oral Qs
Recall: Questions that ask candidates only to recall facts, principles, or other information.
Interpretive: Questions that require interpretation of information, and using principles to deal with new situations.
Problem solving: Questions that require finding a solution.– What is the list of differentials for these findings?– Outline you management of this case.– What would be the best next step in this case?– Can you account for the clinical findings in this case?
McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.
Finding the right balance: prompts, probes
• Plan them in advance but they can never be entirely standardised
• Only plan a few – need to move on to next case/topic
Types of prompts and probes
Exploring the depth of knowledge and higher order abilities
– Can you explain your reasoning?– What are the implications of each of these options?– What is the justification for your decision?– What evidence supports your judgement in this case?– Explain how you arrived at that conclusion– Because?– Why is that?– What would you do if your approach does not work?
Types of prompts and probes
Seeking more detail from the candidate– Any other differentials?– Can you tell me which specific drug you would
use?– Which particular suture pattern would be most
appropriate?– Which particular signs are leading you to conclude
that?– Can you elaborate on that?
Types of prompts and probes
Clarifying the question being asked or redirecting the candidate
– What I am getting at here is…– Can I get you to focus on (x) in this case?– Can you tell me all the options rather than what you would do?– Just coming back to something you mentioned earlier about (x),
can you tell me…
Asking about things that the candidate has missed out– Can you tell me any other implications? What about implications
for the whole farm?– One thing you didn’t mention was (x). Can you tell me about
that?
Discourse
Three different types of discourse in oral exams (Roberts, 2000)• Personal experience
– talk that deals with personal experience and feelings and often involves narratives and stories
• Professional– talk of professionals to clients and colleagues to
colleagues• Institutional
– principles: abstract, analytic, depersonalised.Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375
Discourse
• Important to frame your question in the same discourse that you expect an answer
• Usually you expect an answer in institutional discourse. – Candidates may not know this, especially those
inexperienced in this type of exam, from ethnic minorities or overseas trained.
– Causes candidate uncertainty and can result in differential bias in exam scores
• But in oral exams its easy to slip into one or other of the other discourses
Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375
The following question combines elements of both professional and personal experience.
Question: If someone took out a personal complaint against you, how would you react? Answer: That's where my personal stress management plan comes in . . . .
Although the theme of the question was professional and personal, the candidate responded in an institutional way. Instead of describing how she would feel (personal experience discourse) or what she did in her practice in relation to a specific case (professional discourse), she produced an abstract and analytical statement (institutional discourse) that satisfied the examiners as a response they could mark. Head nodding and mutual smiles provided confirmation.
Quote from Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375
Ancillary materials
• Need to have clear function• Use them purposefully, not as decoration• Ensure they clearly show what they are
intended to show, when displayed in the way candidates will see them– Check with someone who did not see the case
Marking schemes
• Rate on only a few criteria eg– Factual recall– Analysis and interpretation of data– Problem solving– Verbal presentation
• Plus overall rating• Separately grade each main question/topic• Provide descriptors for 3-4 levels of performance
eg: clear fail, marginal fail, bare pass, good pass
Conducting oral exams
Create the right climate• Introductions and non-threatening questions (eg
weather)• Convey a sense of genuine interest• Adapt body language to candidate’s needs
– Some need eye contact, some find it off-putting– Candidate interpretations of “mhm” and “uhuh”:
• Encouragement to go on• Stop talking so I can ask the next Q• Well done you have passed
• Avoid humour
Not to much structure, not too little
• Plan all the questions and possible prompts including how these should be worded
• But on the day don’t read as if from a script• Adapt your language and body language to the
candidate• Change the order or leave out probes if necessary• Use clarification and redirection only if necessary• Give all candidates the same information• Metacommunicate
Dealing with issues
When the candidate can’t answer or says very little• Ensure candidate has had enough time to
respond (but don’t give them endless time)• Do not give hints• Ask the candidate if they would like the question
clarified. If they do, be careful not to– give hints or additional information– change the discourse or requirements of the answer
• Often best just to move on to next question
Dealing with issues
When candidates go off track or take too long• Use clarifying or redirecting questions
– What I am getting at here is…– Can I get you to focus on (x) in this case?– Can you tell me all the options rather than what
you would do?– Just coming back to something you mentioned
earlier about (x), can you tell me…
Dealing with issues
When candidates are exceedingly nervous• Trying to help often makes things worse• Poor performance may be the cause: move on
to the next question
Scoring candidates
• Record grade for each question as soon as it is finished• If you are thinking of giving quite a high grade think
“What would the candidate have to have done to earn an even higher grade?” – helps examiners use the whole scale
• At end of exam record your holistic judgement as well as sum/average of scores
• Do not confer with your co-examiners until you have recorded your own judgement
• Discuss your judgements and reach a consensus decision