Quis custodiet ipsos custodes*?

Enhancement of Supervision and Examination Practices through

Qualitative and Quantitative Methods

Assoc. Prof. Dr. Ing. Luigi VanfrettiKTH SmarTS Labluigiv@kth.se

LH219VStockholm, February 10th 2015.

Outline

• Motivation

• Observations, Challenges and Assumptions, Hypothesis

• Goal of this study

• Previous methodology, survey and sample data available

• What’s next?

• Revisiting Old Practices

• Proposed Methodology

• Enhanced Survey

Motivation: Observationsand Questions

• Some supervisors and examiners (S&Es) think they are great, they are actually convinced about it.

• Are they really as good as they think?

• How have they arrived at such conclusion?

• Would the S&Es self-assessment be valid if:

• If you ask students: might you find a different answer?

• If you gather quantitative and qualitative data: will you see consistent results across different samples?

Motivation: Challenge and Assumptions

• Traditionally, S&Es have evaluated the quality of their teaching

through different methods, e.g. [1,2,3]

• Why is this not an approach also taken to evaluate the quality of

S&E?

• Challenge:

• S&Es are reluctant to evaluate the quality of their S&E

practices.

• Assumption seems to hold [4]

• Assumptions: self-development

• The S/E is a reflective supervisor or examiner.

• This S/E would be willing to use quantitative and qualitative

methods and tools to asses its practice

• This S/E would be willing to apply a systematic method for both

data acquisition, analysis and implementation of changes into

his own practice

Luigi does not, in any way, suggest that this should be even considered to be adopted for assessing

supervisors and examiners. I’m doing it for myself! Let me do my thing…

Motivation: Hypothesis

• Hypothesis:

• S&Es are “willing and able” to enhance their practice

• To do this, they will need to assess their performance to pin-point strengthens and weaknesses

• And, using their strengths, devise a plan for for continuous enhancement (and to fulfill their role

properly!)

• Therefore, a method and tools to achieve this are necessary.

• In general, these methods and tools should be able to aid in the following process:

• Reflection and Data Acquisition: Do I suck as a supervisor/examiner?

– I.E. What are my strengths and weakness as an S/E.

• Assessment: In what and how badly?

– How can I measure (quantitatively and qualitatively) the quality of my work?

• Action: Can I do something about it (systematically)?

– How can I change and how can I document these changes?

• Repeat: Closed-loop control loop for continuous enhancement.

Motivation: Goal

• Goal:

• To derive a new methodology for assessment and an enhanced survey that can be used in the

future by those supervisors and examiners that are interested in enhancing the quality of their

practice.

• To share the experience I’ve gained through a first iteration of applying such systematic

method.

• Background:

• I started this project a few years ago, when I was in the tenure track process at KTH.

• My skills as a supervisor and examiner where put into question by senior staff

• I needed evidence: to be left alone and do my thing

• Consider this specially when your thing is different and works better compared to ‘the way things have always been

done around here’.

• Critical when someone else has control over your own research funds

• I needed evidence to protect myself - so many things can go wrong in the tenure track

• If someone questions your practice, you can answer with facts

• But overall, I needed evidence to become better at what is supposed to be my real job!

• Of course, this is irrelevant for tenure or a full-professorship, nobody cares if you are a good

supervisor/examiner “as long as you show them the money”!

Previous Method

The previous method I used was the following:

1. Adaptation and implementation of a student survey (which included quantitative and qualitative questions)

• Survey originally proposed in [5]. Changes made to adapt for own needs.

• First implementation July 2011. Used Google Forms.

• Second July 2013. Changes to broaden breath (more student types (e.g. BSc, Research Visitors, etc.) + clarify ambiguities.

2. Execution of the survey by the students• At different stages of MSc or PhD or other.

3. Analysis of responses and open discussion during research group meetings

• Not documented. Plan to hold one for the updated survey.

4. Implementation of changes.

Quantitative Data

Qualitative Data

In depth qualitative assessment.

Explanation and Instructions

Aspects Considered in the Survey

Clerical:• Initial guidance and thesis contract/proposal

• Paperwork and bureaucracy

• Easiness in booking meetings with the supervisor/examiner

• Compliance with harassment policy

• Equal opportunities practices related to gender, race,

nationality, etc.

Quality of Supervision:• Availability for continuous supervision and progress

feedback

• Time/value of the supervision meetings

• Provision on practical guidance, methods, etc., in early

stages

• Value and quality of feedback orally or written

Preparation of the

Supervisor/Examiner:• Provision of literature and relative “currency” of the

supervisor/examiner knowledge on published work

• Level of ability/training of the supervisor/examiner

Social and Ethical Aspects:• Social/psychological/emotional/motivational support

from supervisor/examiner

• Scientific standards and good practice of the

supervisor/examiner

• Ethical standards of the supervisor

• Level of advice, support and academic freedom at

each project stage

Dissemination:• Opportunities provided for dissemination and

discussion of research results within technical settings

• Opportunities for exchange in research events,

seminars, lectures or others within the domain of the

project or general field

Execution of the Survey

• The survey was to be filled up at least 2 times a year by PhD students, and at least 2 times by MSc students or others.

• For the MSc students, they would fill up the survey at around 50% of the completion of their project, and a second time at the completion of their project.

• Nevertheless, the implementation of the survey became more seldom in 2014, actually stopping. This was due to Luigi burning out…

• An additional execution of the survey will take place in February 2015, which will be used in the preliminary data analysis.

• Towards the development of a new methodology and enhanced survey, a “preliminary” version of the new survey will be executed following early feedback.

Some Sample Quantitative Information(2011-2013)

Project Progress Degree Information

Completed Projects

Some Sample Quantitative Information(2011-2013)

Some Sample Qualitative Information(2011-2013)

• From a MSc Student: Sometimes you don’t necessarily suck, even if everyone tells you that you suck.

Quantitative and qualitative data will show if there is consistency with other students,

or if this was an outlier case.

A good supervisor may change whole the future of his students.

Since now, I have attended many evaluation meetings, and I always criticized the style of supervision here. The Msc

students have a supervisor for their last semester and not the whole period of study, and no one care for their records

and improvements.

The supervisors are not that much intimate, and they usually don't support students in an adequate manner.

However, I found Luigi to be a different one. He is so motivated, fresh, and full of passion for the job. He works till late

in the night, and even during the weekends just to organize and manage everything in the best possible way. He is so

warm with his students, lets them to express themselves, and provide the necessary freedom for the students to

choose their own style. He asks a lot from the students, and gives them lots of opportunities and chances instead.

…

I love his academic style, and feel extremely comfortable in the research group. He is very intelligent, respectful, and

transfers the motivation to the students also. If it was not due to some limitations, he may perform even better.

Some Sample Qualitative Information(2011-2013)

• From a Male MSc Student:

I experienced a big change during the Thesis. At the start I was not able to work alone and required help always.

I had not my own criteria in order to perform the work.

Luigi really gave me the opportunity to work by myself and taught me how to choose wisely and how to work in

an accurate manner.

So I would like to highlight that I grew personally and educationally while working in the project.

In addition, such an important commitment forced me to be constant and responsible for my work as I had to

report about it.

What I really think was a outstanding was the meetings with Luigi and Tetiana, it was incredible how they were

able to propose new solutions and they came up with ideas. At the beginning I was a little lost but while the time

was going on we improved the ideas and we achieved great results.

Finally, I would to say that having such a nice team as the SmartSLab provides a lot of support and always I

needed help I had it.

• From a Female MSc Student:

He seems to treat women kindlier than men, which is good for me (since I'm a girl) but not for the guys, I believe.

Perception might vary, while the quantitative and qualitative data will show if there is any consistency

in my practices.

Some Sample Qualitative Information(2011-2013)

• From a Female PhD Student:

After having been under Dr. Luigi’s supervision for about one year, little by little, I have come to realize several important things which have significantly influenced my view of the project I am currently conducting. With the guidance from Dr. Luigi, I started to have a clearer picture and a better understanding of the topic; the important problems in my field, the relevantissues, the directions I should be aiming at, and how to tackle the problems. By gradually being questioned, I have developed a ‘self-questioning’ habit and a thinking process – train of thought – in which I have learnt to connect thoughts systematically.

Additionally, having been working with Dr. Luigi, I have realized the different (academic) levels of standards, of which I have now complied with the one thought to have high and international standard, and academic morality. By observing him, I have realized the difference between a regular researcher and a very good researcher, in which he is the role model, and what made him became one of those.

One challenge faced under Dr. Luigi’s supervision is that he always comes up with (many nice) new ideas, to catch up with them and become more independent can be difficult. However, after discussing with him, he agreed to loosen it up. Up to the present moment, I am satisfied with the situation and his supervision.

Two crucial attitudes a supervisor should have, in my opinion, are a commitment to the role and a will to improve him/herselfto become a better supervisor. I have seen these attitudes from Dr.Luigi; he often asks my opinion about his supervision attentively listens to what I say and is open to comments as well as suggestions. He is willing to change his supervising style if it is believed to improve the situations.

In summary, I value the experiences I have had with Dr. Luigi; I have learned many different aspects from him, even from the mistakes, both by my own and by his own. Last but not least, I started to have ‘fun’ with the project which I did not have it before, thanks to Dr. Luigi.

Accepting to make changes will improve your student’s feelings about the project, and they will even

start having fun!

2014 - No data (from Feb.)2015 – Some preliminary data

Quantitative:

Only 8 responses out of 22 expected. Too early to show figures that are meaningful.

Qualitative:

Students notice the difficulties you face (response from 2013)• I am really sad because he is suffering a lot from the stup** hierarchy system here, and he is always forced to do things

that he doesn't want. He is working in a very stressful situation…

Some things are now worse – but the students want to give good data to help fixing things• From this question onwards, the grades reflect my personal perception of the last year. I'm totally aware of your situation,

and the consequences it may involve on your supervision of the different students, therefore it is not in an attempt of criticizing, but more for helping you determine areas of possible improvement, as I noticed you were these past month making a strong effort in getting better and more present.

Some things are still good• The freedom that you enjoy while working under the supervision of Dr. Luigi Vanfretti is extra-ordinary. He will give you

some basic ideas during discussions and will allow you to explore it. I was indeed blessed to have such a supervisor and iam sure i wont be able to have such a Boss in my life again.

So… what’s next?Revisiting Old Practices

• What’s next?

• Revisiting Old Practices

– Revisit the method and the survey.

– Perform data analysis – determine how the survey has to be analyzed (which methods?) to extract weaknesses and strengths.

– Check for ambiguities in survey.

– Carry out and document an open workshop with the students that remain in the research group.

• Important to determine how the examiner’s “health” affected the students, and what to do about it.

So… what’s next?QaQ Techniques and Methodology

• Methods

– Investigate which quantitative and qualitative analysis methods could

be best suited for the available data

– Make modifications to the survey in order to gather data more

accurately

• Proposed Methodology

– Do the survey again, with updates.

– Define a new method using the results of the survey and by applying

the methods.

– Assess the results from the methods through a set of repertory grid

interviews as a cross-validation test.

• Enhanced Survey

– Will make available, together with analysis, in a publication.

Thank you!

Questions? Constructive Comments?

Want to help? Or buy me a beer? Get me a date?

luigiv@kth.se

Good luck with the rest of your day!

(mine is already hopeless...)

References

[1] L. Vanfretti and M. Farrokhabadi, “Consensus-Based Course Design and Implementation of Constructive Alignment Theory in a Power System Analysis Course”, European Journal of Engineering Education, 2014.

https://dx.doi.org/10.1080/03043797.2014.944101

[2] L. Vanfretti and M. Farrokhabadi, “Evaluating Constructive Alignment Theory Implementation in a Power System Analysis Course through Repertory Grids”, IEEE Transactions on Education, vol. 56, no. 4, pp. 443 – 452, Nov. 2013.

DOI: http://dx.doi.org/10.1109/TE.2013.2255876

[3] L. Vanfretti and Federico Milano, “Facilitating Constructive Alignment in Power System Engineering Education using Free and Open Source Software,” IEEE Transactions on Education, vol. 55, no.3, August 2012, pp. 309-318.

DOI: http://dx.doi.org/10.1109/TE.2011.2172211

[4] S. Taylor, “Managing postgraduate research degrees”. In H. Fry, S. Ketteridge and S. Marshall (eds) The Effective Academic: A Handbook for Enhanced Academic Practice. London: Kogan Page.

[5] S. Taylor and N. Beasley. A handbook for doctoral supervisors. Routledge, New York, 2005.