Experiences on Course Development Practices in a Power System Analysis Master Course a.k.a. tales of a journey through the valley of death in providing a high quality learning environment Assoc. Prof. Dr. Ing. Luigi Vanfretti and Maxime Baudette KTH SmarTS Lab [email protected]KTH Scholarship of Teaching and Learning Stockholm, March 12 th 2015.
Transcript
Slide 1
Experiences on Course Development Practices in a Power System
Analysis Master Course a.k.a. tales of a journey through the valley
of death in providing a high quality learning environment Assoc.
Prof. Dr. Ing. Luigi Vanfretti and Maxime Baudette KTH SmarTS Lab
[email protected] KTH Scholarship of Teaching and Learning Stockholm,
March 12 th 2015.
Slide 2
Acknowledgment There has been a lot of former students involved
in this efforts, through teaching assistant duties and beyond,
which we cant list here but you know who you are, and we thank you
very much! A very special thanks is expressed to Mostafa
Farrokhabadi, who was the first student helping with the research
work related to this talk until Maxime Baudette got involved.
Slide 3
Outline Motivation Constructive Alignment Theory
Implementation: T&L activities, Assessment Activities
Evaluation of the implementation: Feedback, R- SPQ-2F, Ranking
Algorithm Filtering Feedback: Repertory Grid Introducing Peer
Instructions with Clickers
Slide 4
EG2100 in Brief Introductory course on Power System Analysis 6
ECTS (7.5 ECTS before 2014) Given on 2 periods (September to
December) About 85 students The course is divided in five important
topics: Modern Power Systems with an Introduction to Sustainable
Energy Technologies and Smart Grids Fundamental Principles for
Power System Analysis AC circuits Electrical Modeling of
Generators, Transmission Apparatus and Networks Methods for
Analysis and Design of Power Networks in Steady State and
Unbalanced Operation Steady State Stability Analysis using the
Power-Flow Formulation Methods for Analysis of Power Distribution
Systems in Steady State
Slide 5
Motivation The previous course (until 2010) that was replaced
by this one had several drawbacks in its pedagogic design and
material: The course material was outdated, inconsistent and lacked
appeal. No teaching method was used to encourage the learning
process The failure rate of the course was very high These
drawbacks required a complete overhaul of the course: from
preparing new material, to the design of a new implementation using
teaching methods. I never teach my pupils, I only attempt to
provide the conditions in which they can learn. A. Einstein
Slide 6
How to address a large variety of background? Students come
with a very different background, and often lack any exposure to
power systems at all: Mandatory for M.Sc. Program in Electrical
Power Engineering (TELPM), and M.Sc. Joint Program in Smart
Electrical Networks and Systems (SENSE) Elective in other M.Sc.
Programs at the school of Electrical Engineering and some Erasmus
Mundus M.Sc. Programs Also offered to power and energy
professionals within industry collaborations and PhD students
(including those from other universities) This variety in the
students background makes it challenging to design well aligned and
adequate T&L activities and proper assessment methods. Students
Background
Slide 7
Redesign of the Course - Goals In 2011 it was decided re-design
the course to achieve the following goals: Increase the success
rate Largely increase the depth and moderately the coverage of the
course content Link the course to the current developments in the
industry and academia, including the Smart grid concept Re-Design
Approach: Implement the constructive alignment theory (CAT) as the
didactic teaching approach of choice for the course Adopt the
consensus-oriented decision-making (CODM) model to design the
course Develop effective multiple feedback channels to gather the
students perception and select relevant feedback
Slide 8
Constructive Alignment Basic Principles (Biggs and Tang) Basic
principle: The theachers fundamental task is to get students to
engage in learning activities that are likely to result in their
achieving the desired outcomes in a reasonably effective manner.
What the student does is more important in determining what is
learned than what the teacher does. Constructive Alignment Theory:
Provides a set of principles These principles can be used to
devising Teaching and Learning Activities (T&Ls) The activities
can help in achieving the Intended Learning Outcomes (ILOs).
Slide 9
Using Principles of Constructive Alignment Using the principles
from constructive alignment: Start by carefully aligning T&LAs
and Assessments These activities should to support the students to
fulfill the ILOs. The students role (MAJOR): The students use the
activities to construct their knowledge and achieve desired
outcomes. The teachers role (minor): To design a learning
environment that encourages the student to perform the T&Ls
that aid the students to construct their knowledge. Course
Objectives (Intended Learning Outcomes) Teaching and Learning
Activities Assessment What should the students do to demonstrate
that they reached the objectives? What work is appropriate for
students to do to reach the ILOs? What should the student be able
to do as a result of the course?
Slide 10
Variety of Background Affects the proper implementation of CAT
Affects students readiness to engage in the Teaching and Learning
Activities Diagnostic test results show a low average in the
students preparation Fundamental knowledge varies a lot. Question %
of correct answers 199 240 390 470 553 684 759 862 969 1027 1132
1284 1369 1453 1577 1690 1767 1863 1967 2074 2144 2268 2351
Total64.8
Slide 11
CODM Design Process (Hartnett) Links the design group to the
consensus model. Five 2-hour sessions (10 hrs) Allows to deliver a
shared proposal: Important if the design group will also be the
implementation group (needed commitment). Closing Framing the
Problem Open Discussion Identifying Underlying Concerns Developing
Proposals Developing a Preferred Solution Choosing a Direction
Slide 12
Design Rule for CAT Implementation Example of using the
consensus based model Step 1 (Framing the problem): design rule
used for one of the T&L activities using Cause-and-Effect
analysis.
T&L Activities - Lectures Lectures are intended to develop
conceptual understanding:
Slide 15
T&L Activities- Invited Lectures Invited Lectures are
intended to develop the interest of students by providing insights
from the industry and top researchers from Academia. Sample of
Invited Lectures Magnus Danielsson. System Design, R&D, Net
Insight AB, Stockholm, Sweden. Lecture on Communication Issues in
Smart Grid Applications Svein Harald Olsen, Statnett SF, Oslo,
Norway. 2 Lectures on IEC CIM Sonja Berlijn, SVP R&D, Statnett
S, Oslo, Norway. Lectures active R&D projects (Lean Line +
Voltage upgrading) David Petesch/Yannick Fillon, RTE - National
Center for Grid Expertise, Paris La Dfense France Lecture on EMTP
and Real-Time Simulation at RTE Prof. Federico Milano, UCD
Challenges for power system modelling and simulation
Slide 16
T&L Activities- Daily Reading Quiz, Conceptual Quiz In
total, they are worth 10 BONUS points of the final grade. Reading
Quiz consists of some very basic questions about the content of the
lecture The questions are designed in a way that a student should
go through the lecture content before arriving to the lecture! Peer
Instruction Implementation Conceptual Quizzes asked about the
lecture content during the lecture. The questions addressed
different learning levels, focusing on developing deep learning.
They sometimes involved a small amount of numerical calculations.
The students answered the questions using a remote clicker.
Slide 17
T&L Activities Homeworks For the students to practice the
methods and study the concepts covered during the lectures. There
are 3 Homeworks (30 points) covering the contents learned in the
respective lectures. The homework are aimed to prepare the students
for the assessment tasks (tests and final exam). In 2014 computer
assisted assignments were introduced. Classroom Home!
Slide 18
Assessment Tests and Final Exam 2 Tests during the semester.
Final exam The tests and exam were built in two parts, with theory
questions in the first part and problems in the second part. The
problems were similar to those solved in the homework. The
assessment of the course is done though the different graded
activities of the course. All the grades are added to give a total
grade on 100 points Course ActivityPoints Daily Quiz10 (Bonus)
Homeworks30 Tests30 Final Exam40 Total100 (+10 Bonus points) Total
PointsGrade Letter 94 100A 87 93B 81 87C 74 80D 65 73E 60 64Fx 00
59F
Slide 19
Evaluation of the implementation Three Course Evaluations One
after each test (end of P1 and end of P2) One after the final exam
R-SPQ-2F Evaluation (at the beginning of the course) Two different
question types (surface and deep approach) Students are graded
using their answers The results will reveal the students approach
Interviews using the Repertory Grid Technique A form of structured
interview to find out a participants preference on a given topic
and the way these preferences are ordered on a rating scale Allows
to link qualitative to quantitative feedback.
Slide 20
Qualitative DataInstructions on Form Explanation on KTH Social
Quantitative Data
Slide 21
Sample Feedback using Google Forms Quantitative Feedback
Qualitative Feedback
Slide 22
Course Evaluations 2011-2014 After 2011, the form was improved
for including an introductory part with instructions, explaining
the students the grading scale they should use.
Slide 23
Course Evaluations 2012-2014
Slide 24
Course Evaluations - YOY improvements The feedback provided by
students that retook the course shows that the changes improved the
course. The acceptance of the changes comes after the students
realize that it has a positive impact on their learning. Note: very
reduced population for statistical analysis Sample Question Is the
new course structure (lectures, daily exercises, weekly exercises,
and weekly tests) preferable to the previous course structure
(lecture, assignments, and final examination)?
Slide 25
Activity No. 1 Design of a course evaluation process
Instructions: Discuss in small groups (no larger than 3) the
following questions. Using your answers, propose a design to
evaluate a sample course. What feedback do you need for your
particular course? T&L activities: do they contribute to the
students understanding? Do the T&Ls have an impact on the
Assessment Activities? In what form do you want it and how much?
Quantitative, Qualitative, Both How do you gather it in an
efficient way? What tools do you use. How often do you gather it?
Number times and type of feedback to be gathered (a) that allows to
trace changes, (b) of new information (originating from the
changes). How do you use it? How do you show the students that
their opinion is considered.
Slide 26
The Revised two-factor Study Process Questionnaire: R-SPQ-2F
Allows to determine the students learning approach The process is
exposed to some drawbacks!
Slide 27
R-SPQ-2F: A new ranking algorithm Deep Learning Surface
Learning 1 st Quarter 2 nd Quarter 3 rd Quarter 4 th Quarter
Slide 28
Grade distribution and its correlation with students learning
approach
Slide 29
Filtering Feedback: Repertory Grids The repertory grid method
is a person-centered approach that allows the student to develop
their own constructs using their own perception of the subject.
Repertory grids serve as a feedback mechanism where the students
can assess course elements using well-defined constructs and
considering different scales. The construction of each grid allows
a methodological analysis of the grids to identify constructive
feedback to improve the course: a formal quantitative method to
select and discard feedback. Good GridBad Grid Good Feedback
Further Analysis Bad Feedback Discard
Slide 30
Implementation of repertory grids The elements of the grid were
determined by the T&L activities. The constructs were built
from interviews with some students A small group of students (about
10%) was carefully selected to represent the different learning
approaches Each student was interviewed individually about the
different elements (qualitative data collection) The constructs
were identified by analyzing all interviews An empty grid was sent
to each student who should fill it in
Slide 31
Repertory grid analysis Different software provide methods for
analysis of the grids (Rep IV, Idiogrid), such as: Principal
Component Analysis (PCA): allows better interpretation of feedback.
Cluster Analysis: re-orders elements in a tree, similar objects are
placed together. Allows to interpret the feedback. Descriptive
analysis (Mean, Mode)
Slide 32
Activity No. 2 Feedback Analysis Instructions: Assume that you
have a high volume course (many participants). In this activity,
you should analyze the method for feedback analysis presented, and
propose alternatives to relate the students to their learning
approach and classify which feedback to address in a systematic
(reproducible) way. Are there other alternatives (methods) to
analyze the course feedback and relate it to the students learning
approach? How do you check if your implementation of CAT is
working? How can you use the proposed method to filter the
different feedback and choose which to address?
Slide 33
Peer Instruction
Slide 34
Peer Instruction and the use of Clickers Reading Quiz and
Conceptual Quiz: To implement peer instruction there is very good
technology available These are called clickers and are used for
gathering students feedback in real-time. Clickers were given to
each student and kept by them throughout the course.
Slide 35
A flexible tool for different uses! In the EG2100 course,
clickers are used for the following: Unique identifier for each
student Attendance registration during Lectures Pre-requisite test
at the beginning of the course (not graded) In-class questions:
Tests on the mandatory pre-readings (5 questions) Questions during
the lecture (~5 questions) Rewarded by bonus points
Slide 36
In-class interactions Tests on the mandatory pre-readings (5
questions) Questions about the content of the pre-readings Regular
questions, all done in a row Questions during the lecture (~5
questions) Questions about the subject being taught Regular
questions, often one by one If average results are obtained,
students are invited to discuss with their neighbors and answer the
question once again (peer instruction) If bad results were
obtained, the lecturer re-explained the material.
Slide 37
Preparation of Teaching Materials for Peer Instruction Material
has to be prepared to introduce the peer instruction as a change of
activity. The lectures need to be prepared in 10-15 min modules
Questions need to be prepared to assess deep learning (or surface
when needed) Instead of a review at the end of the lecture, peer
instruction allows for review of deep learning concepts during the
lecture. This has a cost: Your time! Think if you can afford it
before doing it!
Slide 38
Pros/Cons Pros: Greatly appreciated by the students: Higher
attendance and better preparation of the students Better in-class
attention Helps the teacher to get direct feedback (e.g. unclear
notions) Flexible ! Cons: Additional Data Management Technical
issues may happen in class (crash, bad reception, empty battery,
etc.) Different lecturing style (might be hard to accept by
students in early phases)
Slide 39
Students feedback about the clickers The feedback received is
very positive and encouraging for continuing this peer-instruction
implementation The mandatory pre- readings helped your
understanding of the course material. It was helpful to discuss
your answers with your classmates (peer instruction) The daily
quizzes helped you keeping up to date with the course materials. It
was fun to use the clickers. It helped you be more attentive in
class
Slide 40
Activity No. 3 Peer Instruction Method Using Clickers
Instructions: Assume that you have already your teaching prepared
from previous course deliveries. In this activity, you should
analyze how to modify the course materials to adapt it to the peer
instruction method using clickers. Discuss the following questions,
provide a proposal on how to implement peer instruction and an
example slide for your particular course? At what learning depth
level do you intend to use peer instruction? When using the peer
instruction method, what factors do you need to consider so that
you can use clickers to obtain quick feedback of the students
understanding of a particular topic? How do you transform (timing,
amount of material and content) your course material to adapt it to
the peer instruction method?
Slide 41
Highlights and Words of Caution A wider variety of stakeholders
can participate in CODM process this would allow for a very good
course design addressing different points of view! CAT is a good
vehicle to enhance the students learning. Its implementation has to
be considered carefully. Implementing CAT is extremely expensive DO
NOT BELIEVE the propaganda that says the contrary! R-SPQ-2F new
ranking algorithm efficiently classifies the students learning
approach depth. Useful for sampling feedback from students with
different learning approaches. The two main challenges for the
course designers are: 1)To continuously modify the course so the
surface approach students move toward deep approach. 2)To do carry
out (1) when there is absolutely no benefit to the teaching staff
(a lot of work that brings no reward)
Slide 42
Discussion (1/2) Continuity and Effort Continuity: The teaching
staff changes from year to year depending on the availability of
PhD students (Luigi is always involved and the only one to provide
continuity) Course planning based on consensus requires continuity,
in order to focus on improvements instead of challenging the whole
structure Effort: Workload and Time The T&L activities require
a large workload for the teaching staff to prepare and correct the
assignments (very expensive and not properly funded) Course
planning based on consensus requires time and involvement from the
teaching staff The feedback process requires a lot of time for
gathering and analysis of the results Integration of clickers in
the lectures requires a lot of time (very expensive!) in order to
provide a good experience for the students (questions should be
well organized and well formulated)
Slide 43
Discussion (2/2) Systemic factors: brace yourself to a large
inertia and resistance to change Students and other faculty will
oppose different didactic approaches to those that they are used
too you will face a large resistance to change. Not having a single
exam to be able to pass the course has been criticized and brought
up to the program coordinator year after year This attitude is hard
to change in the 4 th year. However, we have shown that the
acceptance of the students improves over time when they are able to
reap the benefits of a good learning environment. You will waste
your time with complaints from the student union, program
coordinators, department head just for trying to do things
differently (I would say BETTER!) There is no means to enforce
student feedback at KTH, this weakens statistical analysis and the
amount of samples. Changes in rules or flexibility is needed if KTH
wants to take education seriously. No pain, No gain? More like lots
of pain, and no gain! The one most important thing we have learned
is that there is no reward for good teaching in KTH do it at your
own risk and for your own self development. The only benefit you
could obtain, by investing even more time, is to try to generate
some publications for your teaching portfolio (in case you want to
find somewhere else to teach with quality).
Slide 44
Even though this has been a wonderful adventure in teaching and
learning, all good things must come to an end. Luigi cant afford to
continue to subsidize the delivery of the course by using time from
his PhD students (which translates to use the funds from research
projects!) to deliver the course. A loss of 290 000 SEK was
generated in 2014. A good learning environment comes at a cost: we
do not get paid enough to deliver a high quality learning
environment. Luigi will not longer be the responsible for this
course nor involved in it, and with that goes the 4 years of
experience But at least we enjoyed the walk through the valley of
death, and we have learned a lot about how to deliver a high
quality learning environment! Epilogue A tale just like that of the
one ring
Slide 45
Thank you! It's really hard to design products by focus groups.
A lot of times, people don't know what they want until you show it
to them. Steve Jobs Questions? [email protected] Good luck with the
rest of your day!
Slide 46
References [1] L. Vanfretti and M. Farrokhabadi,
Consensus-Based Course Design and Implementation of Constructive
Alignment Theory in a Power System Analysis Course, European
Journal of Engineering Education, 2014.
https://dx.doi.org/10.1080/03043797.2014.944101 [2] L. Vanfretti
and M. Farrokhabadi, Evaluating Constructive Alignment Theory
Implementation in a Power System Analysis Course through Repertory
Grids, IEEE Transactions on Education, vol. 56, no. 4, pp. 443 452,
Nov. 2013. DOI: http://dx.doi.org/10.1109/TE.2013.2255876 [3] L.
Vanfretti and Federico Milano, Facilitating Constructive Alignment
in Power System Engineering Education using Free and Open Source
Software, IEEE Transactions on Education, vol. 55, no.3, August
2012, pp. 309-318. DOI: http://dx.doi.org/10.1109/TE.2011.2172211
[4] S. Taylor, Managing postgraduate research degrees. In H. Fry,
S. Ketteridge and S. Marshall (eds) The Effective Academic: A
Handbook for Enhanced Academic Practice. London: Kogan Page. [5] S.
Taylor and N. Beasley. A handbook for doctoral supervisors.
Routledge, New York, 2005.
Slide 47
Course Objectives (Intended Learning Outcomes) After completing
the course, the students should be able to: Using first principles
derive the basic concepts and methods used for power system
analysis. Describe the methods used for modeling and simulation of
different power system phenomena, and Explain how computer methods
affect the modeling process Construct mathematical models for
computing the steady state performance, and basic unbalanced
performance of power systems. Describe the use of the Unified
Markup Language for data modeling of power system components, and
interpret the Common Information Model and its relation with the
power system models used in class. Describe, derive, and compare
different models of the most common equipment used in power network
models. Using different methods, to compute, analyze, and reflect
on the performance of a power system under steady state and
unbalanced operation, for transmission and distribution networks
Apply a power system and general purpose software tools to analyze
the performance of a power system, and evaluate it for different
operating conditions Contrast different software development
methods that are used in the creation of power system analysis
software Using first principles derive the basic concepts and
methods used for power system analysis. To construct data models
and mathematical models for computing the steady state performance,
and basic unbalanced performance of power systems. To derive,
describe and compare different models of the most common equipment
used in power network models. Using different methods, to compute,
analyze, and reflect on the performance of a power system under
steady state and unbalanced operation To describe basic
characteristics of renewable and distributed energy sources, as
well as monitoring, analysis and control technologies used in Smart
Grids.
Slide 48
Evaluation Diagnostic test at the beginning of the course (non
graded) Raise student awareness about their individual level of
required background In-class questions are rewarded with a very
flexible bonus point system Technical issues require to have a
flexible system Any evaluation activity requires to maintain a
database of students