‘Generic Learning Outcomes’ as a strategic tool for evaluating learning impact Introduction In this paper I will first introduce the Generic Learning Outcomes – GLOs for short, and the parameters within which we use them for evaluation at National Museums Scotland. I will then give a case study example of how we applied the GLOs to the evaluation of one of our exhibitions, and conclude by looking at the strengths and weaknesses of the GLOs and how we plan to take this framework forward in the future. The GLOs provide a common language for colleagues, funders, evaluators and policy-makers to talk about learning. The framework was developed by the UK Museums, Libraries and Archives Council (MLA) to help demonstrate the overall impact that museums, archives and libraries have on people's informal, lifelong learning. The GLO framework recognises the multiplicity of learning as a wider concept beyond the mere acquisition of facts, and comprises five areas:

• Attitudes and values – which can e.g. include perceptions, attitudes or opinions towards

other people, or increased motivation or tolerance • Skills – which can e.g. include physical skills, communication skills or information

management skills • Knowledge and understanding – which can e.g. include learning facts or information,

making sense of something, or making links between things • Enjoyment, inspiration, creativity – which can e.g. include having fun, being surprised or

being inspired • Activity, behaviour, progression – which can e.g. include what people do, have done or

intend to do There is no hierarchy between these areas, and all are considered equally important. More information can be found on the MLA’s Inspiring Learning for All website at www.inspiringlearningforall.gov.uk

Figure 1 – The five areas of the GLO framework (modified from the Museums, Libraries and Archives Council). So how do we use the GLOs in our exhibitions and the subsequent evaluation? At the beginning of any new exhibition, the relevant exhibition group establishes the learning outcomes for the exhibition using the GLO framework. Besides being a team building exercise that ensures everyone is on the same page, this exercise sets out what visitors are intended to get out of the

exhibition, i.e. the ways in which we anticipate it will impact on them. The exhibition team then uses this to guide the development of the exhibition, i.e. look at what possible methods of presentation and interpretation can deliver the desired learning outcomes. We subsequently use these same learning outcomes to define our evaluation and help us to focus on what questions we should be asking to measure what impact the exhibition has had on visitors’ learning. Each visitor will have an individual and unique experience and thus individual learning outcomes – i.e. you can have five visitors looking at the same object in the same space, and they will all come away with a different experience – but the GLO framework enables us to categorise these individual responses and so provides a tool for quantifying qualitative data.

Figure 2 – The GLO planning and evaluation process at National Museums Scotland Case Study

I would now like to talk about an example of an exhibition in which we used the GLO framework to focus our evaluation. At National Museums Scotland, as in many museums in the UK, museum education has grown beyond just being a learning provider for schools, to including informal and lifelong learning for adult and family audiences, and the case study I have chosen was an exhibition aimed at an adult audience. Beyond the Palace Walls was a special exhibition showcasing Islamic Art from the State Hermitage Museum. With around 200 works of art the exhibition highlighted the beauty of Islamic Art and the diversity and complexity of the Islamic artistic tradition. The National Museum of Scotland is undergoing a major redevelopment project. As part of this, 16 new galleries will be created, two of which will be very similar to Beyond the Palace Walls in that they will present beautiful art objects from world cultures in a contemplative setting. We therefore wanted to know what kind of impact Beyond the Palace Walls made on visitor learning in order to inform the development and interpretation for these two new permanent galleries. The methods we used for our evaluation included a self-completion survey with 200 visitors, and pre- and post-visit interviews with 20 visitors, and we also recorded visitors’ dwell time in the exhibition. The questions we chose both for the survey and the interviews were guided by the learning outcomes that had been identified for the exhibition. Around two dozen desired learning outcomes had been identified for the exhibition across the five GLO areas. Alongside regular questions pertaining to visitor demographics, reasons for visiting and overall satisfaction with the exhibition, we had devised questions specifically related to the five GLO areas and some of the questions straddled more than one area:

Attitudes & Values A sample learning outcome for this area was: “Challenge preconceived ideas about Islamic art” The main question we chose was: “Have your perceptions of Islamic art changed after visiting the exhibition” – and the follow up question: “How have your perceptions changed?” And some sample responses from visitors were: “I used to think Islamic art was crude and unimaginative” and “I didn’t know they appreciated pictorial work with animals” Both responses demonstrate a change in perception, and are in immediate relation to the learning outcome of challenging preconceived ideas. Overall, 40% of respondents said their perceptions of Islamic art had changed This question also gave us feedback on some of the learning outcomes in: - Knowledge & Understanding and - Activity, Behaviour, Progression Knowledge & Understanding A sample learning outcome was: “Islamic art has been influenced by Western and Eastern artistic impulses and vice versa” The main question we chose was: “Is there anything you know now that you didn’t know before your visit to the exhibition?” And some sample responses were: “Chinese blue glaze was copied from Islamic ceramics” and “I learned about strong artistic links between East and West” This was not only new information the visitors hadn’t known before, but is also in immediate relation to the learning outcome of the cross-cultural influences between Eastern and Western artistic traditions. Another sample learning outcome was: “Increased awareness of the extent and diversity of Muslim cultures around the world through their art” The same main question gave the following sample responses: “The great extent of the Islamic empire and how much land it covered” and “That there was a large Muslim population in China” Again, this new information that the visitors did not know before their visit is in immediate relation to the corresponding learning outcome. We deliberately avoided asking visitors ‘what did you learn’, as from past experience we know that questions phrased like that put visitors off, because it makes them feel like they are at school. Overall, 58% of respondents said that after their visit they knew something new they had not known before their visit. This question also gave us feedback on:

- Attitudes & Values and - Enjoyment, Inspiration, Creativity Enjoyment, Inspiration, Creativity A sample learning outcomes was: “Appreciation of artistic skills and talent” The main question we chose was: “Was there anything in the exhibition that surprised you?” And a sample response was: “Some of the intricate detailing and workmanship” This was not just something that had surprised the visitor, but is also in immediate relation to the learning outcome of appreciating artistic skills and talent. Another sample learning outcome was: “Enjoyment of exhibition design” The same main question gave the following sample response: “The use of lamps and screens giving a calm and gentle feel to the area” Again, this element of surprise for the visitor is in immediate relation to the corresponding learning outcome. This question also gave us feedback on: - Knowledge & Understanding Activity, Behaviour, Progression A sample learning outcome was: “Increased interest in Islamic art” The main question we chose was: “Has the exhibition inspired you to find out more about Islamic art?” – and the follow up question “How will you find out more about Islamic art?” And the response was that 39% of respondents were inspired to find out more about Islamic art, e.g. by browsing the internet, research at the library, visiting other relevant galleries at the museum, buying books, travelling or visiting a mosque. Unproven Learning Outcomes However, there were also some learning outcomes we could not prove: In ‘Attitudes & Values’ 1 out of 4 learning outcomes was unproven: “Muslim visitors may make connections to Islamic cultures other than their own” This was not due to the methodology, but to the fact that the proportion of Muslim visitors was less than 1% and thus did not pose a statistically viable sample from which to draw conclusions. In ‘Knowledge & Understanding’ a total of 6 out of 8 learning outcomes were unproven. An example of one such outcome is: “Muslim cultures and their art depend on the cultural, social, political and religious situation of the areas they were created in”

Again, this was not so much due to the methodology, but to the nature of the exhibition and the interpretation of the displays. All of the learning outcomes for ‘Enjoyment, Inspiration, Creativity’, and the majority of learning outcomes for ‘Attitudes & Values’, were achieved, which was largely due to the fact that they did not rely on reading large amounts of text. The relevant impact could be made simply by looking at the displays and minimal reading of object labels, which gave information on name and origin of what was displayed. It became evident from the general feedback we were getting that it was a very aesthetically pleasing exhibition, so it is not surprising that learning outcomes based largely on visual impact were easily achieved. Most of the learning outcomes for ‘Knowledge & Understanding’, however, relied on visitors reading text panels with larger amounts of information. From the feedback we got on visitors’ general satisfaction with the exhibition, we know that 72% wanted a bigger variety of ways to engage with the displays and the subject matter, e.g. maps, contextual images and photographs, audio and film. So the fact that we did not cater to different learning styles among our adult audience had a direct negative impact on their learning. Finally, an example of a learning outcome for ‘Skills’ was: “Being able to recognise certain symbols and motifs of Islamic art” As you may have noticed, ‘Skills’ did not feature in my earlier examples. There were only two learning outcomes for ‘Skills’, but we could not prove either of them. Both involved learning to recognise something, in this case symbols and motifs, and short of making visitors undergo a flashcard test we were not able to measure this through survey and interview questions. A further issue was that while e.g. a change in perception or attitude can be fairly instantaneous, most learning – including being able to recognise certain symbols and motifs – happens over time. Which brings me to my conclusion. Conclusion To summarise, the strengths of the GLOs are that they provide a common language for staff working together both within and across organisations, to talk about learning. They can help to steer projects so that the focus on the end user, i.e. the visitor, remains a priority. And subsequently, they can help to focus evaluation by ensuring that relevant questions are asked, thus giving a more accurate measure of the impact projects make on visitor learning in relation to the desired learning outcomes. However, there are also weaknesses in this approach. Some learning outcomes, especially in ‘Skills’’, are difficult to measure without putting the visitor through a test. Furthermore, most learning happens over time, and what sticks in visitors’ minds immediately after their visit may not necessarily be retained. Finally, in relation to visitor behaviour and progression, what visitors say they will do only measures immediate intent and relies on visitors’ self reflection. What this approach does not measure is the long term impact of learning. Depth interviews with visitors both before and after their visit, rather than just exit surveys, are a step in the right direction in terms of measuring the change in visitors’ meaning making. What we are piloting for our special exhibition this year, however, is a longitudinal study in which we will be following up with visitors several months after their visit to the exhibition to find out what they still remember and if they have followed up on any of their planned further actions. We hope this approach will give us a more accurate measure of the long term impact of exhibitions on visitor learning. References Museums, Libraries and Archives Council – Inspiring Learning for All: www.inspiringlearningforall.gov.uk Jenni Fuchs, Visitor Studies Officer National Museums Scotland j.fuchs@nms.ac.uk