Date post: | 01-Jan-2016 |
Category: |
Documents |
Upload: | gwendoline-irene-horton |
View: | 214 times |
Download: | 0 times |
GLAD NSS RESEARCH PROJECT
‘I can’t believe it’s not better’: The Paradox of NSS scores for Art & Design
David Vaughan and Mantz Yorke
This research was sponsored by the ADM Subject Centre of the Higher Education Academy and by the HEAD Trust
Aim to help colleagues to:
• Better understand why the subjects receive the ratings on the NSS that they do; and
• Identify how best they can respond to expectations and the enhancement of the student experience
Detailed data from the 2007 NSS:- broadest JACS grouping of 19 subject areas; and - the finest grouping of 107 subjects(+ further analysis of 2008 and 2009 data and student pre-entry qualifications/experience)
Interviews with the seventeen key informants + a focus group to:
- gain an insight into how, and to what effect, institutions have responded to NSS outcomes
- perceptions of the factors affecting student responses to the NSS
The paradox:
- Creative Arts & Design make a substantial contribution to the national economy
- It is a national success story
- Yet when it is viewed through the lens of the NSS, it does not come out well in comparison with other subject areas
Our ‘take’ on the NSS:
It has been validated statistically
However, it is flawed- conceptually (including not suiting all subjects/HEIs equivalently)- technically- it mixes up inputs and outcomes
Crudely:Inputs = T & L; A&F; Academic Support; O & M; Learning ResourcesOutcomes = Personal Development; Overall Satisfaction
Our ‘take’ on the NSS.....
It seems to suffer from a ‘halo effect’- most scales correlate highly with Overall Satisfaction
Is it a valid representation of the students’ overall experience?- the late Malcolm Eley’s research on the Australian CEQ suggests not
The NSS is here to stay (for the time being, at least – but watch out for instruments focusing on student engagement, as in the US and Australia)
So we have to live with it, even if we don’t like it
And we have to admit that even if it’s a gross ‘measure’, we can still get messages from it
A (semi-)permanent fixture
Messages:
Creative Arts & Design perform relatively weakly in the NSS, save in Assessment & Feedback
40%
50%
60%
70%
80%
90%
100%
T&L A&F AcSup O&M LRes PersDev OvSatis
Med/DentAllied MedBio SciVet Sci etcPhys SciMath&Comp SciEngineeringTechnologiesArchit etcSocial studiesLawBus&Admin StMassComm & DLinguistics etcEuro Langs etcEast Langs etcHist&Phil StudCreative A&DEducation
Creative Arts & Design do relatively poorly (NSS 2008)
Percentage agreement
2009 data not available in this form
Surridge (2009, p.18)
Reference subject area: Law
JACS subject area T&L A&F AcSup O&M LRes PDev OvSatis
Medicine and Dentistry +/ns/ns -/-/- -/-/- -/-/- +/+/+ +/+/+ ns/-/-
Subjects Allied to Medicine ns/ns/+ ns/ns/ns +/+/+ -/-/- +/ns/+ +/+/+ ns/ns/ns
Biological Science ns/-/ns +/ns/ns +/+/+ +/+/+ ns/ns+ -/-/- ns/ns/ns
Veterinary Science ns/ns/ns -/-/- ns/ns/ns -/-/- ns/ns/ns ns/ns/ns -/-/-
Agriculture and Related -/-/- ns/ns/ns +/ns/+ -/-/- -/ns- -/-/- -/-/-
Physical Sciences +/ns/ns +/+/+ +/+/+ ns/ns/ns +/+/+ +/ns/ns +/+/ns
Mathematical Science -/-/- +/+/+ +/+/+ +/ns/+ +/+/+ -/-/- ns/ns/ns
Computer Science -/-/- -/-/- +/ns/ns -/-/- +/+/+ -/-/- -/-/-
Engineering and Technology -/-/- ns/ns/- +/+ns -/-/- +/+/+ -/-/- -/-/-
Architecture, Building and Planning -/-/- ns/-/- ns/ns/ns -/-/- ns/ns/- ns/ns/- -/-/-
Social Studies -/-/- +/+/ns ns/ns/ns -/-/- -/-/- -/-/- -/-/-
Business and Administration -/-/- ns/+/ns ns/ns/ns -/-/- ns/+/+ ns/ns/ns -/-/-
Mass Communication -/-/- ns/ns/- ns/ns/ns -/-/- ns/ns/ns -/-/- -/-/-
Languages +/+/+ +/+/+ +/+/+ ns/ns/ns -/-/- ns/ns/- +/ns/ns
Historical and Philosophical Studies +/+/+ +/+/+ +/+/+ +/+/+ -/-/- ns/ns/ns +/+/+
Creative Arts and Design -/-/- +/ns/ns ns/-/- -/-/- -/-/- -/-/- -/-/-
Education -/-/- +/+/+ ns/+/ns -/-/- -/-/- +/+/+ -/-/-
Combined ns/ns/ns +/ns/ns +/ns/ns ns/ns/ns ns/ns/ns -/-/- ns/ns/ns
Two subjects -/-/- +/+/+ +/ns/ns -/-/- -/-/ns -/-/- -/-/-
Three or more subjects -/-/- +/+ns +/ns/ns -/-/- ns/-/ns ns/-/- -/-/-
Three strikes…
3 yrs above av 3 yrs below av
Messages.......
Creative Arts & Design perform relatively weakly in the NSS, save in Assessment & Feedback
Institutions offering major Creative Arts & Design programmes vary widely in their NSS scores (as will be seen later)
There may be internal explanations for why the data are as they are, but these won’t be appreciated by the general public
Disaggregating to JACS 107 subjects:
Percentage agreement T&L 2007, selected subjects (JACS 107)
0
20
40
60
80
100
Percentage agreement A&F 2007, selected subjects (JACS 107)
0
20
40
60
80
100
Percentage agreement AcSup 2007, selected subjects (JACS 107)
0
20
40
60
80
100
Percentage agreement O&M 2007, selected subjects (JACS 107)
0
20
40
60
80
100
Percentage agreement LRes 2007, selected subjects (JACS 107)
0
20
40
60
80
100
Percentage agreement PerDev 2007, selected subjects (JACS 107)
0
20
40
60
80
100
Percentage agreement OvSatis 2007, selected subjects (JACS 107)
0
20
40
60
80
100
Disaggregating to institutions with studio-based provision:
Overall Satisfaction, Creative Arts & Design 2007
T&L Asst & Fdbk
Org & ManAcad Sup
Learn Res Pers Dev
Regarding Learning Resources, Creative Arts & Design seem to do a little less well than laboratory-based subjects
The vertical bars here represent the range of % agreement in the various HEIs
Interview survey:
- NSS clearly here to stay - results taken seriously
- Results a “wake up call”
- Some staff rested on laurels - “we are better than others so we don’t have a problem”
- None complacent or defensive
- More a sense of frustration
- Majority of results at best weakly positive • esp. Organisation & Management and Assessment &
Feedback
- Despite actions such results continue
Consideration of NSS:
- Now part of annual monitoring - alongside other evidence
- Detailed action planning
- Internal surveys in advance or alongside - same or similar questions
• often similar messages but NSS scoring/comments generally harsher
• stronger pressure for action from external NSS survey?
Actions taken by institutions:
- Emphasis on actions to enhance student experience
- Communicate actions better through dialogue
- Regular feedback through websites - focus groups - meetings
e.g. student reps taking “tea with the Dean”
- Poster campaigns - “You said it – We did it”
- Realistic marketing/prospectus information:
• what is promised is delivered
• student expectations realistically informed
• student support services reviewed - increased or more focused provision
Underlying NSS concerns:
- How students in Creative Arts interpret questions • view that the pedagogy of Creative Arts & Design does
not sit well with questions • more for highly timetabled lecture-based subjects
Underlying NSS concerns.....
- Questions not designed to draw out • how personal development supported• whether a well-rounded educational experience received
- Important to reflect on the student experience based Creative Arts pedagogy:
• to improve our understanding of it • our ability to explain it
Organisation & Management
- Data since 2005 - consistently lower NSS ratings than other subject areas:
• apparent that the drivers behind the data are complex
• range of institutional factors have a bearing - institutional structures - major or minor changes in the physical environment
- how Creative Arts are located or perceived within the institution
Interview survey comments include:
Often a direct correlation between good course leadership and good NSS scores and vice versa
Importance of better communication with students: – “it’s not rocket science to do things like alerting
students to the cancellation of a lecture or laying down expectations about the availability of part time teaching staff”
– recognised students often confused about who to go to for information
– increased emphasis on information/course administrative services
Interview survey comments include.....
Greater emphasis on management of student (and staff) expectations– “what’s the deal?”– what students can expect of institution– what institution should expect of student
– some revisiting or resurrecting Student Charters
Interview survey concerns included.....
- creative arts much more complex - self-determined and negotiated - than highly timetabled lecture-based courses
- much more forward planning and negotiation required to procure specialist resources to complete a project
Therefore NSS question:
“I have been able to access specialist equipment when I needed to”
positive reaction less likely - equipment had to be booked into/queued for - (in contrast to having it available on demand)
Interview survey concerns included.....
Pedagogy of Creative Arts could mean that:
“the more a student has creative freedom the more they are likely to experience what they believe to be a failure of Organisation & Management”.
NSS question:
“the timetable works effectively as far as my activities are concerned”
could lead to negative comment that:
“the timetable was not effective as it got in the way of their self-determined creative project work”
Many other factors affecting Organisation & Management
but often returned to was the pedagogy of Creative Arts– our understanding of it – our ability to explain it to our students (and to others)
Assessment & Feedback
Assessment & Feedback has consistently attracted low ratings across the whole of HE
Survey comments suggest feedback more of an issue than summative assessment in Creative Arts.
Actions taken include:
Much work undertaken to raise awareness of students and staff: – production of assessment handbooks highlighting
Assessment & Feedback – discussions to explain formative and summative
feedback and encourage engagement
Despite initiatives very little impact on the NSS results: – do staff - as well as students - fully understand
assessment and feedback models in use? – can we assume accepted models work adequately?
Other questions raised:
– do students really understand learning outcomes/assessment criteria - how they are used?
– are statements in handbooks sufficient without further exemplification?
– in our efforts to make things more explicit “have we made everything over-complex and difficult to understand?”
Other questions raised.....
Close working relationship between staff and students can also lead to misunderstandings:
– “I thought you said I was doing OK (feedback) but now I’ve not got the marks I expected (summative assessment)”;
– formative feedback given in passing often challenging/intended to stimulate thinking but can even be seen as an instruction
Other questions raised.....
Feedback often not recognised or understood: – “the only form of feedback students in Art & Design
recognise is that tailored to them as individuals - what do I have to do to get a better mark (i.e. 1st or 2.1)?”
– “Our students wouldn’t recognise they were receiving feedback even if we all went round wearing T-shirts that said You are now getting feedback”
Some reflections...NSS results being viewed and considered very seriously
There appear no general problems with student recruitment or commitment to courses in Creative Arts & Design
Memorable experiences – good or bad – often recent - are likely to have a significant effect on NSS scores and comments
Issues that seem to exercise students the most are:– the stimulation of the teaching– engagement of, and contact with, academic staff– feedback (possibly a need to manage perceptions)– Organisation & Management of programmes– access to specialist resources/equipment
So... Why the paradox? Who’s at fault? Fact or anecdote?
Special pleading will not be taken seriously
Why are graduate surveys often much more positive?
Pedagogy of HE Creative Arts & Design:- Do we take it for granted?
- Do we explain it?
- Do we understand it?
- Do we do enough to prepare our students for HE study?
- Do we know what our students expect/expected? - We probably need to ask them but not through the NSS
The full report is available on the HEA-ADM website:
http://www.adm.heacademy.ac.uk/projects/adm-hea-projects/national-student-survey-nss-project