Received 15 March 2013Received in revised form27 June 2013Accepted 23 July 2013

Keywords:Humancomputer interface

d learning objectshis shift has beenand learners, have

s need to competels of usability andenough to exper-

Over the last four years we have been involved in the development of the HumBox, a digital repository for the Humanities educationalcommunity in the UK. First developed in 2009 the HumBox was launched in February 2010 and has developed a small but active usercommunity (with around 900 registered users, sharing around 1500 open educational resources). The HumBox was developed throughuser-centered design, and in particular co-design guided by principles from user-experience (UX) and ergonomic design, with the aim ofenabling innovation. The vision was to create a website that was not only usable and scalable, but which supported this relatively small but

* Corresponding author. Tel.: 44 (0)23 8059 5567; fax: 44 (0)23 8059 3218.E-mail address: dem@ecs.soton.ac.uk (D.E. Millard).

1

Contents lists available at ScienceDirect

Computers & Education

Computers & Education 69 (2013) 287302www.humbox.ac.uk.In the last ten years research around digital learning content has changed dramatically, from a focus on richly describeheld in managed environments to open educational resources (OER) existing within a broad ecology of web systems. Tdriven by a new generation of Web 2.0 collaborative and sharing systems, that while not specically aimed at teachersbeen enthusiastically embraced for teaching and learning.

Supporting a community of subject teachers and learners within this new context is challenging. Technical systemwith commercial sharing sites such as YouTube and SlideShare, with users increasingly demanding professional levescalability. Yet it is still important to provide these smaller communities with digital places where they can feel secureiment, share their practice and grow their community.

11. IntroductionLearning communitiesOpen educational resourcesLearning object repositories0360-1315/$ see front matter 2013 Elsevier Ltd. Ahttp://dx.doi.org/10.1016/j.compedu.2013.07.028the last three years the challenge for HumBox has been to act as not only a shared library for its com-munity, but also to change the working practices of individuals in that community by encouraging themto work in a more open way and to actively share their materials and ideas with others. Getting users toengage with and adopt innovative systems is a well-known problem; with the HumBox our approachwas to focus on user-experience (UX) design through agile development and ongoing participatory andco-design activities. In this paper we present a mixed-methods evaluation of the success of thisengagement over the past three years, focusing especially on the way that users have appropriated thesystem and its services in order to solve real problems. Our evaluation reveals that for many HumBoxusers we have been successful in creating a technology that is invisible (meaning beneath the level ofnotice and concern) and that the result of this is less micro-appropriation (users adopting and usingspecic features in new ways) and more macro-appropriation (users adopting and adapting the site as awhole). We conclude that in the case of HumBox invisible technology coupled with the social frameworkof co-design and user engagement activities, has allowed a diffusion of ownership, and created a safesocial and technical environment where the community can debate high-level issues, and that this hasled to changes in both professional and pedagogical practice.

2013 Elsevier Ltd. All rights reserved.Article history: The HumBox is a learning resource repository for the Humanities educational community in the UK. OverThe HumBox: Changing educational practice around a learningresource repository

David E. Millard a,*, Kate Borthwick b, Yvonne Howard a, Patrick McSweeney a,Charlie Hargood a

a School of ECS, University of Southampton, Southampton SO17 1BJ, UKb LLAS, University of Southampton, UK

a r t i c l e i n f o a b s t r a c t

journal homepage: www.elsevier .com/locate/compedull rights reserved.

growing community with very low entry barriers and exible tools; the ultimate goal being to enable the sharing of experience within thecommunity leading to individuals changing practice.

1.1. Research objectives

In this paper we present a mixed-methods evaluation of these goals, focused on the use of the HumBox website and the engagement of

and its community of users, and explores how this behavior relates to the design and engagement methodology undertaken by the

ference accounts (The Utopian Years? a collection of presentations given at a half-day conference on Pompidous France), and professional

www.heacademy.ac.uk.4

D.E. Millard et al. / Computers & Education 69 (2013) 287302288www.eprints.org.5 http://www.elanguages.ac.uk/l2o.php.6 http://www.elanguages.ac.uk/claret.php.7development materials (The Pool a collection of resources to support the professional development of English teachers, in areas such ascourse design and assessment).

1.3. Structure

Our paper is structured as follows. Section 2 describes related work in the area of learning objects, OER and repositories, as well assetting out the design approach taken in HumBox, which is grounded in theories of user-experience design, ergonomics and innovation.Section 3 then presents the mixed methods used in our evaluation (numerical analysis, user survey, and in-depth formal interviews) anddescribes how they map to our three research goals. Section 4 presents the results of each part of the study, alongside an initial inter-pretation of those results. Section 5 is a cross-study analysis that draws out a number of high-level conclusions in regard to the appro-priation of the site by the community and the resulting changes in practice. Finally Section 6 concludes the paper and summarizes ourndings.

2 http://www.jisc.ac.uk/whatwedo/programmes/elearning/oer.aspx.3development team. This is in contrast to existing analyses, which have tended to take a more quantitative high-level view across a range ofexisting systems, or undertaken stakeholder analysis on larger OER repository sites (these are discussed further in Section 2.1).

1.2. HumBox

The HumBox is an online space for the publication and sharing of open educational resources relating to the study of the humanities inUK Higher Education. Membership of the site is open to all and is entirely voluntary. It was created as part of the HumBox Project which wasfunded under phase one of the JISCs OER programme.2 The project was a collaboration between 11 different HE humanities departmentsacross the UK and 4 Higher Education Academy3 Subject Centres. It was led by the Centre for Languages, Linguistics and Area Studies, basedat the University of Southampton. The aim of the project was to encourage humanities practitioners to publish and share their teachingresources as open, adaptable content, and to engage actively with the processes and issues related to open practice. The HumBox repositorywas created to facilitate and enable activity on the project.

The HumBox was designed and developed by a team within the School of Electronics and Computer Science, at the University ofSouthampton, and is a bespoke version of the successful ePrints4 research repository (also developed at Southampton). Both the technicaldevelopment of the site and the community engagement processes practised in the project built on earlier knowledge and experiencegained under previous University of Southampton-led JISC- and EduServ-funded projects: L20,5 Claret,6 MURLLO,7 Faroes.8 These projectswere linked by a common theme: the attempt to understand the needs of a particular discipline community (in each case, languageteachers) when approaching and engaging with open practice for the rst time and to realize this understanding through the developmentof a user-friendly teaching and learning repository that would foster the integration of open practice with day-to-day academic life. Each ofthe afore-mentioned projects saw a new iteration of the repository design as developers responded to user feedback in working to achievetheir aim (Borthwick, Arrebola, Millard, & Howard, 2009). The HumBox would become the latest version in this evolutionary chain ofteaching and learning repositories.

Fig.1 shows an example HumBox resource page (more about the HumBox design can be found in Section 2.4). HumBox resources includea wide range of media and cover both in-class and self-study activities; for example, they include information sheets, presentations, videointerviews, audio conversations, resource lists, and photographs. While there is no typical resource, examples include student contributions(such as the Romanian Thoughts an illustrated podcast completed by a Masters Student that reects on post-Ceausescu Romania), con-its community. More formally our research objectives are:

1. To measure and model users engagement with the HumBox site.2. To measure the impact of HumBox on their beliefs, behaviors and practice.3. To assess the success of the design process, especially in regard to users appropriating the technology in innovative ways.

Our methodology is described in more detail in Section 3, which also explains how the various data collection techniques we have usedmap to these objectives.

Our work is novel in that it provides a quantitative and qualitative view of behavior around a specic teaching and learning repositoryhttp://www.elanguages.ac.uk/murllo.php.8 http://www.ecs.soton.ac.uk/research/projects/454.

D.E. Millard et al. / Computers & Education 69 (2013) 287302 289Fig. 1. A HumBox Resource Page multimedia previews rather than metadata are the focus of the pages in HumBox.2. Background

The work on HumBox is part of a trend in e-learning toward open educational resources, partly inspired by the early work of pioneeringinstitutions such as MIT and their OpenCourseWare9 initiative, OpenLearn10 from the Open University in the UK, and grassroots onlinecommunities such as Connexions.11 The UK government in particular has put signicant funds into exploring and promoting open practice inUK HE.12 However, the challenge of sharing content between e-learning systems, and in teaching with resources designed by another personat another institution, goes back before this to earlier work on learning objects and educational repositories.

Our work in HumBox has focused on reinventing learning objects for a Web 2.0 world, and in particular on working closely with ourcommunity, using principles from user-experience and participatory design, to create a website that would enable innovation in thebehavior of our users. In this section we look more closely at the history of learning objects, and the related work in design theory andmethods, before explaining the conceptual framework we used in our own approach and its roots in theories of innovation.

2.1. Learning objects and repositories

Within e-learning there has been a long standing tradition of describing and sharing open educational resources (OERs), most notablythrough the use of learning objects (LOs), collections of stand-alone teaching materials that are given a formal metadata description inorder for them to be searched, organized and reused, for example IEEE LOM (IEEE, 2005). Collections of LOs are usually made availablethrough learning object repositories (LORs) (Neven & Duval, 2002), software systems that store LOs and typically make them availablethrough a Web interface. LOs and their repositories therefore succeed or fail together, depending on both the quality of the LO content andalso the schemas, interfaces and tools in the repository (Downes, 2001).

Learning objects have not been as successful as many hoped; it has been argued that the reasons for this include the time and technicalskills required to construct and package LOs (Bratina, Hayes, & Blumsack, 2002), and the difculty of translating and capturing pedagogicalpractice (Freisen, 2004; Polsani, 2003). In our ownworkwe have found that users needmotivations beyond pure altruism in order to engagewith learning repositories, for example providing tools that provide valuable services such as le conversion and web hosting, relegatingsharing to a secondary function (Millard, Howard, McSweeney, Borthwick, et al., 2009).

There has been very little work focusing specically on design methodologies for LORs, or qualitative case studies of their use, but at-tempts have been made to survey and categorize different approaches, and these contain lessons for design. In particular, Neven and Duval

9 MIT OpenCourseWare: http://ocw.mit.edu/.10 OU OpenLearn: http://www.open.edu/openlearn/.11 Connextions: http://cnx.org/.12 www.jisc.ac.uk/oer.

(2002) present an early overview of LORS, and identify benets of particular design decisions, some of which (such as personal workspaces,user proles and automatic metadata generation) are elements that informed the design of HumBox. Tzikopoulos, Nikos, and Riina (2009)present a later survey of LORs that reects a maturing of the eld as they identify wider issues such as quality control, cost, and IPR ap-proaches (all issues faced by HumBox), but they do not suggest how these might t into a greater design philosophy that could help resolvethem for a given community. Ochoa and Duval (2009) present a quantitative analysis of 39 LOR systems, surveying their composition anduse over time. Their analysis leads them to conclude that there are broad classes of LOR that have different characteristics, and the designersmust be clear over their stakeholders in order to plan how to incentivize them correctly (something that was crucial to the social aspects ofHumBox); they also point out that LORs behave in a similar way to other user-generated-content, and therefore existing infometrics anddesign principles might apply.

As an example, the key problem faced by LORs of encouraging stakeholders to contribute is one that is shared with many interactive websystems, a challenge that Fischer (2009, 2011) describes as fostering cultures of participation. Fischer advocates a holistic approach thatincludes stakeholders as rst-class citizens, and in particular advocates co-design based around authentic activities. This is analogous to theapproach we have taken with HumBox.

OpenLearn from the OU is an example of a large OER repository with an active and varied community. OpenLearnwas originally designedwith two distinct sides: LearningSpace, where learners can use social and personal tools to manage their online learning, and LabSpace,where teachers can contribute and remix materials (McAndrew, Santos, & Godwin, 2007). Collaborations within OpenLearn tend to beinstitution driven, with motivations including access to existing content, online tools, and afliations with the OU itself (McAndrew et al.,2009). HumBox has similar goals to LabSpace, but with a smaller community who are engaging as individuals or small teams, rather than atthe institutional level in this context fostering a culture of participation is particularly challenging.

Connexions is another successful site where users can collaborate to create custom courses from open content. Initiated by RiceUniversity but hosting resources authored by a wide variety of contributors, Connexions is notable in that it takes digital textbooks,and digital publishing as the metaphor for interaction, in that users can assemble collections of materials into virtual course texts thatothers can follow (Baraniuk et al., 2002). It also encourages post-review of resources, and provides lenses where users can lter poorlyor non-reviewed materials (community reviews were an important activity for the HumBox community too, but were neverformalized in this way). Later studies have shown that Connexions users participated in order to increase their impact within theirdiscipline (Baraniuk, 2008), and thus attribution and visibility was a prerequisite for contribution. It also meant that a positiveConnexions brand was a key to participation, particularly in ensuring that the brand captured the progressive values of the underlyingproject. This is very similar to our own experience with HumBox, where the overall project became an important symbol for thecommunity.

In terms of a similar technology Kresimir, Natasa, and Hrvoje (2010) take a similar approach to us in adapting an existing repositoryplatform (in their case FEDORA), but the focus in their work is in the integrationwith an existing LearningManagement System (LMS) and asa result many of the challenges they describe are technical (such as data exchange and LO composition) rather than community or design-focused.

More recent trends in the area of OER and LOR research have lead to the development of MOOCS (Massively Open Online Courses), whichextend the notion of creating and releasing OERs into developing fully edged public online courses that can potentially be accredited(Daniel, 2013). MOOCs represent an alternative to traditional models of face-to-face education, and have even been perceived as a threat totraditional educational institutions and professionals (Vardi, 2012).

Connexions, OpenLearn, and many of the existing LORs work similarly to MOOCs in that they typically hold coherent collections ofmaterials representing complete resource sets for a course, which give an inherent structure to student-learning even if students canpackage resources in their own unique collections. However, HumBox does not work in this way, rather it holds a diverse range of resourcesthat are precisely not structured or complete. HumBox is therefore not focused on nalized learning objects (polished, high quality, digitalresources which students access as they are), but rather on practitioners themselves, and notions of remixing, adaptation, and the sharing ofideas (Busetti, Forcheri, Ierardi, & Molno, 2004).

To achieve this sense of open sharing we could not depend on institutional engagement, and instead had to appeal to individuals. Thishas caused us to look at replicating the success factors for existing community content sites, and in particular using methods from user-experience design and drawing on theories of successful innovation.

2.2. User-centered and user experience design

The perspective of the user has long been considered a key aspect of software design, and was fore-fronted in User-Centered Designmethodologies such as participatory and co-design, where users are brought into the design team to contribute to and shape softwaredevelopment (Grudin & Pruitt, 2002). In our own previous work we have looked at how these principles can be extended into a theory andpractice of co-deployment (Millard, Howard, Gilbert, & Wills, 2009).

Over the last decade there has been a move away from purely functional design, to considering the full user experience (UX). Hassenzahland Tractinsky (2006) describe UX design as being about technology that fullls more than just instrumental needs in a way that ac-knowledges its use as a subjective, situated, complex and dynamic encounter.

As such UX Design considers aspects of the user experience that are more nuanced, sometimes requiring an inter-disciplinary approach(Sengers, 2005). For example, Picard (1995) argues for an understanding of emotional aspects, Mahlke (2005) considers hedonics andaesthetics alongside affection and emotion, and Nor and Muhlberger (2010) explore how it might be possible to design for empathy.

UX Design has been criticized for being ephemeral and lacking in practical methods. For example, Bardzell and Bardzell (2008) discussesthe subjectivity of experience, arguing that it may be impossible to capture something so individual in a general sense, and Nack (2003)describes the difculties in modeling and understanding experiences in dynamic contexts. van Harmelen (2001) goes as far to commentthat a complete set of descriptions can never fully describe the richness of the users world.

However, frameworks and practical methods for UX Design have emerged. Gruen, Rauch, Redpath, and Ruettinger (2002) describe how

D.E. Millard et al. / Computers & Education 69 (2013) 287302290user stories can be used to better understand the experiences of individual users, capturing the perspective of an individual within a given

context. Gaver, Dunne, and Pascenti (1999) used cultural probes to gather information, values and responses from a community, and Dix(2003) describes how experiences can be deconstructed to go beneath surface effects to emotional responses.

UX Design has also been explored within the software engineering process. Naja and Toyoshiba (2008) describe how it can be inte-grated into agile development practices, Buchenau and Suri (2000) developed a methodology of experience prototyping, using paper-basedsystems to test and explore a proposed experience, and Hooper and Millard (2010) describe TAPT, a formal method based around Dixsnotion of experience deconstruction where an existing system is deconstructed and reconstructed in a new context, but with experientialaspects (rather than surface elements) maintained.

2.3. Innovation

While user-centered design is focused on producing systems that are genuinely useful to a group of users, and user-experience design oncapturing non-functional, implicit or tacit requirements, we had an additional need with HumBox in that wewere explicitly trying to designfor innovation creating a system that would encourage its users to change their existing practice.

The challenges of innovation are well known, and were captured by Rogers (1962) in his seminal text the Diffusion of Innovations.Rogers denes innovation as an idea of practice that is perceived as new by an individual of other units of adoption. He also identiedseveral key attributes of innovations that are familiar challenges in software development:

Relative advantage the extent to which a new innovation improves on existing practice Compatibility the degree to which an innovation is compatible with the existing sociocultural values, past experiences and needs ofadopters.

Complexity the degree to which an innovation is perceived as relatively difcult to understand and use. Trialability the extent to which adopters can experiment, at low-cost, with a potential innovation. Observability the extent to which the outcomes of innovation are visible to other potential adopters.

Rosenberg (1972) is critical of the view that innovative ideas are static throughout the adoption process, and argues that there is feedbackfrom the experience of adopters that causes the ideas to evolve and adapt; this ts well with the ongoing conversations seen in co-designand agile development, where a continuous feedback loop is encouraged. Network effects have also been shown to play an important roleduring the adoption process, especially with digital communication technology (Abrahamson & Rosenkopf, 1997). However, while in-the-large innovations can take years or decades to take effect (David, 1990), we believe that with smaller scale digital innovations user-centeredapproaches and agile development can signicantly help.

2.4. The HumBox conceptual framework

Our design and development work in HumBox was based on a co-design strategy and agile development plan. A variety of techniqueswere used throughout the project, including show and tell workshops, tutorials, surveys and face-to-face training and interviews. Inaddition the process was disseminated via national events and supported through the LLAS website and mailing lists.

We were sensitive to UX issues in that we framed the design activity as an ergonomic challenge. Ergonomics, the study of how to t asystem to a person, is perhaps more traditionally thought of in the sense of physical devices and machines, but it can equally be applied todigital systems and applications (Eason, 1991). In particular we have based our co-design activities on theories of cognitive and macro-ergonomics (Falzon, 1990; Hendrick, 1995) where a more holistic approach is taken to understand the context of a system and in ourcase the non-functional UX requirements. Other work in this area has focused on designing new systems or processes, for example in theworkplace (Kleiner, 2006), but we have used it within the design process to t a new activity (the creation and sharing of OER resources)alongside existing activities in as frictionless a way as possible.

From this viewpoint the approach we have taken with HumBox is not unlike the work of Sharples et al. (2002) who developed a socio-cognitive engineering approach that seeks to develop a theory-based framework of users underlying cognitive and social processes as aguideline to design. It is also similar to Normans notion of activity-centered design, where a set of tasks is considered as a whole within aparticular context (Norman, 2005). Both of these have in common a desire to transform the way that users work, rather than just replicatingwhat is already done in a digital system.

The result of our ergonomic approach was that we identied a number of key challenges that seemed to be missed by existing learningrepository approaches. These included:

The importance of delivering a service of value in Rogers terms this is about maximizing the relative advantage to users depositingOER materials. We made no assumptions that the deposit activity would be incentivized or would just happen, but instead identiedservices of value that we could offer to users. In our co-design workshops these were identied as enabling users to get multimediaresources onto the web without the need for expertise (enabled by in-line previews), and supporting lightweight community in-teractions (by enabling web presence through the use of user proles, and attention metadata and interaction through comments andreuse tools).

A strong focus on removing barriers in Rogers terms this is about minimizing complexity. It was clear that existing learning objectstandards were signicantly more complex than the majority of teachers considered necessary, by minimizing the number of elds(from over 40 in the original LO spec to just 4), translating the terminology from that of pedagogical theory to that of practitioners, andfocusing on the content in the existing resource instead of requiring additional content. We also removed content packaging, relyinginstead on the pages, keywords and simple collections to add structure.

Adoption of existing knowledge structures in Rogers terms this is about achieving compatibility. At the project level we encouragedthe publication of open educational resources within a sociocultural context that was familiar to users: focusing on the public aspects of

D.E. Millard et al. / Computers & Education 69 (2013) 287302 291professional work; enabling the sharing of ideas, and inviting review of published work (the innovation in our case would be that these

elements would be applied to teaching resources and those resources would be open and adaptable). At the technological level wedropped the distinction between resources and activities that had been present in our earlier prototypes, and insteadmade everything aresource (that may include materials, activities or a mixture of both). We did this because it was clear that to create object typesrequired teachers to make distinctions that were not normally needed. For example, we identied some resources with an absent butimplicit activity (such as a class exercise), and others where the activity was part of the resource, but was not identied as a separateobject (such as an exam rubric) a phenomenon that we called phantom tasks and invisible rubrics (Millard, Howard, McSweeney,Arrebola, et al., 2009)

The need for a social framework which enables what Rogers calls Trialability and Observability. In developing HumBoxwewere luckyto have a series of projects to support community events and fund the creation of example materials. We were also fortunate to haveenthusiastic champions in the form of the LLAS who were able to go to our users with an existing reputation, giving them social capitalthat encouraged people to become involved. They were also able to use their expertise to identify potential change agents in thecommunity, and to organize and run effective community events.

The result of addressing these challenges was that while other LO repositories made relatively supercial changes inspired by Web 2.0designs (such as adding ratings or tags), or doggedly retained onerous metadata requirements for depositors, we adjusted the core purposeof the site, aiming to create technology that minimized the barriers for sharing.

3. Methodology

The HumBox project has been ongoing for over three years, and during that time both the site (software) and the community haveevolved. This makes an evaluation of the HumBox site difcult. In addition the success factors for HumBox are complex, it was not sufcientto generate trafc, or to get users engaging during the core project, instead we needed to establish the site as an ongoing engagement, andwork toward changing our users attitudes and behavior in regard to OER and sharing their teaching materials.

The three research objectives presented in Section 1 were:

1. To measure and model users engagement with the HumBox site.2. To measure the impact of HumBox on their beliefs, behaviors and practice.3. To assess the success of the design process, especially in regard to users appropriating the technology in innovative ways.

To evaluate these objectives we have taken a mixed-methods approach (Tashakkori & Teddlie, 2003), devising a study with three parts:

A numerical analysis, of the HumBox logs and database (quantitative). This builds a numerical description of the site, its users and theirinteractions (objective 1).

A user survey, of the broad HumBox community (both quantitative and qualitative). This creates a statistical picture of how usersresponded to the site beyond their simple interactions, for example exploring how they incorporated the site into their working livesand how the site inuenced their own teaching (objectives 1 and 2). Qualitative responses also give clues as to the rationale behindthese views and actions, which can then be more fully explored in the nal part of the study (objective 3).

User interviews, of key community members (qualitative) giving us the opportunity to explore in more detail the individual stories ofchanging practice, and to explore the success of failure of our approach in terms of UX design (objectives 2 and 3).

In terms of Leech and Onwuegbuzies typology we would classify our work as a fully mixed concurrent equal status design (Leech &Onwuegbuzie, 2009), as the quantitative and qualitative parts of our study were mixed during both the collection and the analysis ofour data.

In the rest of this section we look at the methodology of each part of the study in turn, and describe the process and parameters of datacollection.

3.1. Numerical analysis

The purpose of the numerical analysis is to build a picture of the HumBox site in terms of the people, resources and interactions over itslifetime. We dene six forms of interaction:

Resources represents a user creating a new page in the repository. A resource is a page with its own url, title and description thatcontains one or more les.

Views represents an HTTP retrieval of a resources HTML representation. Downloads represents a single le downloaded from a resource. Bookmarks represents a user adding a resource to their personal bookmarks list (sometimes this is called favoriting) Comments represents a user adding a single comment to a resource Collection represents a user creating a collection of one or more resources

Data was collected from the HumBox database using the EPrints programming API. Cumulative totals were created by separating itemsinto blocks of months using the date stamp for their published status in the repository. Download and view counts were created by theEdShare statistics module, which processes information from the EPrints access dataset. The access table is ltered for robots and repeatclicks using a list of known robots, meaning that number of views and downloads is a close approximation to actual user totals.

Table 1 shows the data that we have gathered and its purpose. The datawent back to the sites creation (as a beta) inMay 2009, and spans

D.E. Millard et al. / Computers & Education 69 (2013) 287302292thirty-ve months to March 2012 (as we collected the data in April 2012). Note that collections became available on the site in June 2009,

and comments in September 2009. Cumulative data for bookmarks was not available, as the creation date for bookmarks was not logged.

response bias (for example, individuals closer to the project may have been more likely to respond). To check for this we asked some basic

Type of data Data collected Purposedemographic questions that have enabled us to show that the responses come from a range of different types of HumBox user.The results of the numerical analysis are shown in Section 4.1.

3.2. User survey

The user survey was designed to complement the quantitative results by capturing the reasoning behind individuals usage patterns.Face validity of the surveywas ensured through the use of an online survey tool, which automatically laid out and presented the survey in

a manner that was appropriate to the responders browser and device. To ensure content validity we derived six high-level researchquestions from our three research objectives:

How is HumBox used in daily practice? Are there barriers to remixing and if so, what are they? What are the factors that help sustain the community? As a user what is the impact of HumBox on my institution, my community and me as an individual? What are the patterns of use that encourage the widest range of users to contribute? How are the HumBox Web 2.0 capabilities (such as collections) being used?

These high-level questions were then mapped to survey questions that asked about specic behavior within HumBox. Both quantitativeand qualitative questions were used in order to build both a numerical picture of engagement, and to explore issues underlying thatengagement (these qualitative comments also fed into the design of the structured interviews).

The survey was then tested via cognitive walkthroughs with critical friends within the project (who had experience with the site, butwho where not part of the survey team) to check for instrumentation bias (i.e. overgeneralization, ambiguity, leading questions, etc.) andproblematic wording was revised.

We distributed the online survey to all registered users of the HumBox, it was publicized through the HumBox project network andposted on the front page of the HumBox site. The survey was published a year after the launch of HumBox when the site had around 450users who had produced 1400 OER resources. The survey was live for four months.

Given that we were surveying an established community our responses were necessarily opportunistic, this means we may have aCumulative dataover time for.

B Number of resourcesB Number of viewsB Number of usersB Number of downloadsB Number of commentsB Number of collections

To create a numerical picture of how the HumBox user base, resource collection, and interactionswith that collection have grown over time (for example, to see if the site has grownsteadily or in stages).

Frequency distributionof users for.

B Number of resourcesB Number of bookmarksB Number of collectionsB Number of comments

To create a numerical picture of the distribution of interaction types across the HumBox userbase (for example, to see whether there is a dependence on a small group of power users).Table 1Data collected for numerical analysis.

D.E. Millard et al. / Computers & Education 69 (2013) 287302 2933.3. User interviews

Interviewswere conducted with a small sample of registered HumBox users. Twenty users were approachedwith thirteen agreeing to beinterviewed. Users were selected for interview on the basis that they had deposited at least one resource into HumBox at some point in theprevious year (201112). Six of the interviewees had been part of the original HumBox project and seven had joined HumBox voluntarily andwere not part of the original project team.

Interviewees were sent the interview form in advance and in most cases, interviews were conducted by telephone or Skype. Twointerviewees preferred to put their responses in writing on the form itself and declined to be interviewed by telephone. In these cases,clarications and further information on their responses was sought and received by email. Telephone interviews followed the formatof the interview form and verbatim responses were recorded by the interviewer. Responses were analyzed for comments and in-dications of changing practice through the use of HumBox, and how far the use of technical features in HumBox has been inuential inthis process.

4. Results

In Section 3 we described our mixed-methods approach. In this sectionwe present the data generated by the study and identify patternsand trends. Then in Section 4 we analyze and interpret these results in respect to our research objectives.

4.1. Numerical results

Fig. 2 shows the scale of the HumBox site and community over time via a cumulative graph of users, resources, comments and collections.The data reects a relatively small but focused community and shows that at the time we extracted the data (April 2012) the site had 904users and 1531 resources deposited. In terms of secondary interactions, those users had made over 436 comments, and created 145collections.

The HumBox project itself ran from April 2009 until April 2010, with the site ofcially launched in February 2010, during this period aperson was employed to work with partners to deposit resources. In addition in July 2011 a separate project (OER2 OMAC) deposited 55items in a single import. From November 2011 a third project (OpenLIVES) began to promote HumBox as a home for its resources.

Fig. 3 shows the impact of the site in terms of visits and downloads. The site has received 40,311 unique visitors, which is respectable foran academic site of this type, and these visitors seem to be highly engaged, as these visits resulted in 10,896 downloads. One might expectvisits and downloads to reect the number of resources in the site, but instead the number of visits has grown almost linearly, whiledownloads has shown slightly better than linear performance (growing 100% in the last 7 months).

Table 2 shows the data for user interactions (where an interaction is the creation of either a collection, bookmark, comment or resource).Fig. 4 shows the frequency distributions of the same data. In all four cases the distribution excludes the number of users with no activity ofthat type. In addition the resources and comments distribution excludes a top outlier, the projects support person who helped communitymembers upload their materials, and this single account uploaded 507 resources andmade 225 comments (accounting for 33.1% of the totalresources, and 60.3% of all comments).

Following the theory of participation inequality (also known as the 90-9-1 rule) (Nielsen, 2006) while we might expect all of ourregistered users (representing 10% of the total users) to be taking some form of secondary interactions in the form of comments, bookmarksor collections, we might only expect around a tenth of these (the 1%) to be contributing resources. With this as a guide we can see that wehave achieved this level with resources, but that bookmarks, comments and collections underperform. Also as expected the frequencydistributions (Fig. 3) shows a long tail, reecting the fact that a few key individuals make a signicant contribution.

The numerical analysis reveals that the HumBox community extends to over 900 people, but that themajority of the sites 1386 resources,436 comments and 145 collections were created by a kernel of active users representing about 10% of the total community.

D.E. Millard et al. / Computers & Education 69 (2013) 2873022944.2. User survey responses

We had 54 respondents to our survey. Fig. 5 shows their prole in terms of how they discovered HumBox, and whether they havecontributed resources. The survey was sent to all registered users as well as advertised on the website itself, so it is perhaps not surprisingthat themajority of the respondents had a user account. However, of these less than half (42%) had uploaded resources, and only one quarter(26%) were part of the original project, so while our survey may represent some of the most active HumBox users, it still contains a mixtureof contributors.

With our survey wewanted to see how users had incorporated HumBox into their working lives. A key part of this is to understand howthey use the site with their own resources. The breakdown shown in Fig. 6 shows that around half (49% of respondents) had uploaded theirown resources to HumBox.We also asked respondents to indicate inwhich ways that had used those uploaded resources (it was possible foreach respondent to selectmore than one answer that applied); most indicated that they had uploadedworking teachingmaterials, but 9 hadFig. 2. Scale of the site, in terms of cumulative number of users, resources, comments and collections.

D.E. Millard et al. / Computers & Education 69 (2013) 287302 295prepared materials specically for HumBox. It was also clear that HumBox had been used directly, both to support independent study (7respondents) and within a classroom setting (6 respondents).

Fewer respondents had used other peoples resources from HumBox (22%, shown in Fig. 7), but within this small group there was avariety of uses, with the majority editing the resource (7 respondents) rather than using it directly.

Perhaps most encouragingly of all 24% of our respondents said that the resources they had seen in HumBox had had an inuence on theirpedagogic practice, and 10% said they had used one or more resources directly away from teaching (shown in Fig. 8). For example, onerespondent indicated that watching a video on detecting plagiarism had helped with his own practice in the area.

The survey also included a number of qualitative questions exploring how and why people valued HumBox. Many of the responses wereterse, or simply pointed at useful resources, but we manually coded the remaining responses into themes and sub-themes. Table 3 shows asummary, alongside the comments themselves.

A number of respondents commented on how HumBox created an awareness of the practice in other institutions. For some this wasabout seeing common practice around standard topics (such as writing for academic purposes), but for others it was about building anunderstanding of the relative differences between institutions, and ensuring that they did not fall behind. Perhaps most importantly it wasabout taking inspiration from what was happening elsewhere, aided by the inter-disciplinary nature of the content and breadth of thecommunity.

Another theme in the qualitative comments was how HumBox could support teaching, and make new practice possible in places whereteachers felt that they did not have the technical skills, or were lacking in time. This later aspect manifested as using HumBox as aspringboard for the creation of their own resources.

Finally there were a number of comments on how HumBox had changed practice and altered the ways that respondents were thinkingand working. At a supercial level this built on the themes of inspiration and awareness, with respondents picking up and using new ideasand tools. For others HumBoxwas a prompt for self-reection, causing them to question and improve the breadth of techniques used in theirFig. 3. Impact of the site, in terms of cumulative views and downloads.own teaching, and in one case exposing some self-doubt about the quality of their own materials.

4.3. Interview responses

The interview respondents indicated a range of motivations as important in getting them involved with the HumBox: nancial incentiveand/or being part of the original project team; dissemination activities such as talks at conferences; word of mouth, or being a participant inan engagement workshop. No-one came to the site through serendipitous discovery and this seems to clearly indicate that champions ofHumBox have been key in initiating, encouraging and driving activity on the site.

4.3.1. Examples of changing practiceMany respondents reported an impact on their professional practice as a result of engagingwith the HumBox. Their responses reveal that

open practice through the vehicle of HumBox is changing the way that they view themselves as HE professionals:

Table 2Analysis of interactions (collections, comments, bookmarks, resources).

Interaction No. of users withzero interactions

Total no. of activeusers: number (%)

Total number ofinteractions

Range of number ofinteractions per user

Average number ofinteractions per user

Standarddeviation

Collections 872 32 (3.54%) 145 039 0.16 1.65Comments 867 37 (4.09%) 436 0225 0.48 7.65Bookmarks 820 84 (9.29%) 591 039 0.65 3.39Resources 818 86 (9.51%) 1386 0507 1.53 17.8

D.E. Millard et al. / Computers & Education 69 (2013) 287302296It has raised lots of questions for me about sharing information and about the concepts of whether we should be reusing materials from atheoretical perspective are we moving towards a world where we have an identical way of teaching undergraduates? The whole process ofpreparing stuff to go on there [HumBox] made me aware of my pedagogical practice in a way I hadnt before. It meant a scrutiny of my teachingthat I dont usually have this was quite unnerving and this focussed my mind on what I was doing.this was useful because at that time Ithought I wanted to move more into academic development work (which Ive since done).

.it has made me think about some of my attitudes in the production of materials how much my materials belong to me or my university Im not sure Ive come to a conclusion about this. I like sharing and I dont like things locked behind passwords.

All respondents felt that their experience using HumBox to share their teaching resources was positive and benecial for themselves andthe humanities community, but that such open practice was unusual:

I think people are generally hesitant to communicate and collaborate especially when it comes to teaching resources (which are oftenperceived as something to keep to yourself).[HumBox] has shown me alternative ways to do things and more creative means to get there.

Another respondent indicated that involvement with HumBox had made me more aware of the opportunities for collaboration generally,and it was a revelation for another respondent that it could be acceptable to base a lesson entirely around open third party content, thuseffectively collaborating with a colleague or colleagues: I hadnt thought of HumBox resources in this way before: the fact that all you mightneed for a session could be in HumBox, and you just draw it all together and teach it and each individual would use that content in a differentway.

Several respondents noted that they had used their HumBox prole and publications in order to assist them in gaining promotionwithintheir institution or to get a new position in another institution. This is a new way for practitioners to demonstrate professional impact asteachers, as distinct from their academic research work:

Mymain purpose for using HumBox was self-promotion. I was in a research fellow position when I joined and wanted to move to a lectureshipposition. I saw HumBox as an opportunity to advertise my teaching (and research) and make examples.available to potential employers.

Fig. 4. Interaction activity in terms of stacked frequency distributions for bookmarks, comments, collections and resources (capped at max 40 interactions).I referred to things I have published in HumBox in my interview for promotion which I got.

Fig. 5. Prole of survey respondents (number, %).

Fig. 6. Breakdown of how users have used HumBox for their own resources.

D.E. Millard et al. / Computers & Education 69 (2013) 287302 297All respondents reported that publishing their teaching resources resulted in an increased awareness of their potential impact withintheir discipline community beyond their own institution, and with the wider general public.

Its made me more aware of the importance of distributing things to colleagues in other institutions. It has enabled me to have a publicdimension to what I do and to make students aware of this.so HumBox has given me an opportunity.

The acknowledgment that publishing open content gives a public aspect to teaching, has led many respondents to review how theyprepare their teaching materials and to consider more closely issues of quality and how their resources may .communicate beyond myclassroom and my students, for example, one respondent commented that: I am more neat in drafting my activities and I think about otheraudiences apart from my own.People I didnt know have contacted me though seeing my resources on HumBox.

Engagement with the HumBox site has become integrated into how some respondents communicate and present materials to their ownstudents: It is my default place for putting things that I have the copyright of. Im using HumBox to keep my nal versions up to date which I usewith students and I have a link [to it] in the VLE. HumBox seems to facilitate this integration: Students, management and maybe me expectnew modes of learning and teaching, and these have to be integrated and HumBox is easy and good. It is part of the perpetual search forinteresting tools to develop teaching and learning strategies.

4.3.2. UX factors (emotional responses to the site)An interesting aspect of the interviews conducted for this research was the range and depth of the emotional responses to the HumBox

site that were recorded. The sixmembers of the original HumBox project team all agreed or strongly agreedwith the statement I feel I partlyown HumBox and they dwelt on this feeling of ownership during their interviews, for example:

The fact that I was associated with HumBox from its inception meant it was easier for me to change [my practice]. I know that my colleaguesthat use it. dont have the same relationship to it that I have because they are using a product in a way- for which they had no input. The factthat I had input means I feel closer to it. Other colleagues like it but theres no commitment.Fig. 7. Breakdown of how users have used HumBox with other peoples resources.

Fig. 8. Breakdown of how users have used HumBox indirectly.

D.E. Millard et al. / Computers & Education 69 (2013) 287302298I felt I was part of it and it was mine, in a way, so I could discover more uses [for HumBox].

One respondent was even concerned that HumBox might become too successful (in terms of gaining large numbers of users and re-sources) and so we may end up losing [it] as a practitioner tool.

This sense of ownership conveyed by the original HumBox collaborators was also expressed in direct criticism of the site: There weretechnical issues that.slowed me down. Im quite persistent but I know others who would just give up; and in repeated unsolicited recom-mendations for the sites improvement, e.g.: I think more could be made of the tags (a tag cloud on the front page perhaps?); weekly/monthlyspotlight on a particular topic (using the tags as its basis).

The respondents who had joined HumBox more recently did not offer any criticism of the site, although they occasionally offeredrecommendations for improvement or advice on how to recruit more users. However, while they did not express the same sense of site-ownership as the core HumBox group, they did share the idea that involvement with the site meant involvement with a communitygroup: HumBox is a research resource and a teaching resource. I felt reassured by being part of the community because it provides access toinformation by people working in the same area. HumBox can be trusted.better than downloading material from the internet in general andit is very focussed you know you are going to get a set of resources, reading, or materials related to what you are looking for, and I like the factthat with the proles you know who you are getting material from and you can see what their interests are. HumBox does create a sense ofcommunity.

This positive sense of community that HumBox seems to have engendered in the interviewees is supported by further comments fromthose who admitted to intermittent engagement with the site: I think its [HumBox] a brilliant idea. Any lack of engagement is probably myfault, and Im afraid my use of HumBox is more aspirational than anything else i.e., all the stuff I would upload if only my time managementwere better. It is curious to note that these users blame themselves for not being more involved with HumBox rather than criticizingfeatures of the site itself which has failed in some way to draw them back as regular users.

4.3.3. Response to technologyWhen asked to comment on the technical system that is the HumBox, interviewees responded with bland, short, similar responses: it isuser-friendly, if you are normally computer literate, it is easy to upload, tag and describe; easy to use, or HumBoxs simplicity is great. Infact, an emerging theme in the responses from all the interviewees was the sense that HumBox is perceived as a community of people

Table 3Qualitative responses coded into themes and sub-themes.

Theme Sub-theme Comment

Awareness Understandingdifferences

It has been useful to see what other HE institutions are up to which texts they are using etc.

Inspiration A great way of keeping up with developments in your discipline and an excellent source for inspirational ideas.[I like] the sharing element and seeing things from the perspectives of other disciplines

Support Lacking technical skills [on using] online dialect resources, I would have found this difcult to do myself, [as I am] not as technically advancedLacking time [I use HumBox] to look at OERs when researching a topic Im going to teach, to see if there is anything I can use or adapt.

I have used a ppt on Sociolinguistics as a springboard to design resources which are relevant to my students.Changing

practiceNew ideas The recorded lectures of English History were very interesting and are making me think that I need to do the same.

The resources with Articulate have also caught my attention as they look extremely professional and very suitable fordissemination to wider audiences. The Hull Fair collection has given me ideas for my work.It is simply a great way to get ideas and improve your own teaching. Its not effortless but it is remarkably easy and intuitive.[I was inspired by] new styles and formats student interviews and projects

Self-promotion For me as an early career researcher, it is a good way to demonstrate what teaching I have done to future employersand to communicate my research.

Reective practice being able to see what other people are doing because I never feel what I produce is good enough to shareAlso, some of the simplest resources, for instance a set of questions for discussions in seminars or lm analysis (English),or pictures of historic sites or symbols (French) helped me to reect upon my own teaching and I have used more these strategies.

collections as these often bring up resources that I might not otherwise nd, or comments, I would say that commenting is the strongest feature

I recommend [HumBox] on the grounds that it is an interesting resource, but also the whole concept of it is really interesting its potential rule for uploading resources, but not for secondary activities such as creating collections. In addition when asked in our interviews aboutwhich specic features enabled practice change, our participants were unable to identify specics and resorted to generalities (such as thesite being easy to use).

In fact, across our evaluations there was a focus on high-level values. In our survey respondents talked about the advantages ofawareness, supporting the teaching process, and changing practice, rather than site-specic details, and in the interviews participantscontinually took a holistic view: for example, choosing to talk about the debate around IPR, rather than giving a specic response to aquestion about technical features (the technical response would have been about using creative commons licenses on each le). In theinterviews there was also a strong emotional response to the HumBox community and the values inherent in the project, rather than anyinvestment in the technology itself.

Given the importancewe placed on software design, it may seem counter-intuitive that the softwarewas so invisible to users, but this is adirect consequence of playing to Rogers ideas about complexity and compatibility, and in particular our observation at the design stage thatremoving barriers was non-negotiable, which forced us to take a simple approach to both metadata and structures.

Another important factor toward the invisibility of the software appears to be the use of a co-design development approach. Thisengagement has meant that rather than the software being the face of HumBox, it is the co-design team, technology champions and otherusers that are identied with the site, and this has helped elevate discussion from specic technology problems to more holistic issuesaround the place of OER in professional practice.

In our evaluation participants clearly saw HumBox as a social network rst, and repository second. The limited size and constraints onways of organising materials that could be useful to lots of different stakeholders and sharing materials.

Other interviewees were keen that HumBox should be dened in a way that disassociates it from being a straightforward technicalsystem: [HumBox] is not just a repository, an archive. You dont want people to think of it as a repository it is active and teaching materialsshould be living things, and personally I feel that Humboxs strength is in ideas rather than content (although the content is of course veryuseful). This sentiment of HumBox as a concept (rather than a system) was so strong that one interviewee, while commenting that herprimary reason for using the sitewas self-promotion,went on to say that .this is of course not what HumBoxwas intended for. In somewaystherefore I might be said to be exploiting HumBox.

5. Analysis

The evidence in our evaluations suggests that our efforts to develop HumBox have resulted in technology that is invisible to users, and thathas led to changes in practice, although not in the ways that we originally expected.

5.1. Invisible technology

A strong theme to emerge from our evaluation was that the HumBox software was mostly invisible to users. There are a number ofindicators that this was the case. The use of specic features seems limited, for example in the numerical analysis we achieved the 90-9-1on Humbox. However, the majority referred to HumBox in a holistic sense, as a system or a concept: .you give examples [different HumBoxfeatures] and I dont think I could identify with any single one, I think it is the whole package.

Interviewees commonly described the site as .essentially a social networking tool for lecturers, .a website for practitioners or apedagogical tool, and repeatedly returned to their perceptions of the nature of HumBox, rather than the technical site itself:(users) rather than a technical system. This was apparent when asked what aspects of the HumBox system (e.g. tagging, collections etc)enabled practice change? (question 4.2) Many respondents either ignored the question and did not respond, or they responded obliquelyfocusing on open educational practice in general, as for example, in this response: The fact that I was sharing and seeing others sharing. Therewere some issues about copyright that my attention was drawn to that made me think about my materials.the fact that the materials were notcopyrighted was good for education a positive step for education because of free distribution.

Most interviewees needed to have question 4.2 explained and once it was understood they nonetheless responded inways that focussedon aspects other than the technology of HumBox, such as the concept of sharing:

Not much to do with the technology but more to do with the concept. What it comes down to the tagging and looking at the proles isinteresting but the aim of that is just to have a nose around. The technology enables you to have that social side, social interest rather thanhelping you in any pedagogical sense.

Or the community:

The people in and around the HumBox system. I see the humbox system as wider than the site itself we were engaged in the community. Thesystem helped in that it existed, was exceptionally user-friendly, attractive and very web 2.0-y, and having developers make changes as wewentalong was desperately important. The system was the focus for our activity, discussions and practice.

4.3.4. Holistics not specicsInterviewees registered a persistent blindness when asked to focus and comment on the technology of the HumBox site and its various

technical features. This blindness was also manifested in how interviewees described the site in general. Several interviewees mentionedspecic technical features of HumBox that they use, such as collections: When I do a search through HumBox I pay particular attention to the

D.E. Millard et al. / Computers & Education 69 (2013) 287302 299membership also helped, giving users a credible place where they could safely build an identity. While specic features (such as proles and

humbox.ac.uk/533/]. This was an exciting way to teach students about Bradford before a visit but also a way to teach using online re-

In our work on the HumBox repository we took a co-design development approach to tackle this challenge, and were guided by UX and

ergonomic theories, aiming to achieve Rogers key attributes of innovations. Our hope was that by better understanding the contextual,cognitive and emotional barriers we could work to remove them through the design of the software and address them in our interactionswith the community.

In this paper we have presented a mixed-methods evaluation of the HumBox over its three year lifetime, based around a numericalanalysis to create a picture of usage, a user survey to create a statistical picture of the impact of the site, and semi-structured user interviewssources in a fun and interactive way. One day I hope to use this resource as inspiration to create similar resources on other topics

However, another strong theme in both the survey and interviews was that HumBox had helped users reect on their existing practice, interms of both making users aware of what sorts of things were being done and used elsewhere, and also in thinking about their own re-sources beyond their own classrooms to broader audiences.

HumBoxwas also seen as a means of communicating with students, and a source of useful resources that could act as a shortcut for hard-pressed staff, one interviewee commented:

The timetable of an academic today is ridiculous teaching, research, admin.it is not possible to maintain a degree of planning ahead asyoud want to, so HB is very good in time of panic [.] The study skills materials are very useful. This is the sort of lesson I never prepare! So itallows me to survive a survival kit.

HumBox also affected professional practice. A small number of respondents in our survey (10%) had used HumBox resources for pro-fessional interest (for example, to reect on the ways they dealt with plagiarism), but we see a more signicant picture when we considerHumBox as a whole. Both our surveys and interviews revealed people that had used HumBox as a tool for self-promotion (both esteem andimpact) and in one example had used their engagement with HumBox in a successful case for promotion. Interview participants also talkedabout HumBox in terms of broadening their digital literacy, making reference to nding out about newmodes of teaching and learning, andalso spoke about HumBox helping them to appreciated the public dimension to their teaching, both in terms of contributing to a largercommunity, but also engaging with the debate around OER.

We specically set out to design HumBox to be exible and easy to use to enable users to appropriate it into their existing practiceeasily. However, we expected this to be at the level of specic tools or features, what we would call micro-appropriation (for example,developing particular tagging strategies, or adopting collections in a specic way). Whilst this did happen in a small number of cases, theoverwhelming story from our evaluation and our interviews is that people adopted the whole site in ways we did not imagine, what wemight call macro-appropriation. So for example, we saw the site used for self-promotion, or as an organizational tool, or for personaldevelopment.

Given our observations about invisible technology it is tempting to say that our focus on complexity and compatibility enabled this, butthis relationship is not entirely clear. Instead our participants seemed to associate this macro-appropriation on the strong social frameworkaround HumBox, describing their ownership of and commitment to HumBox, and the trust that they have in the HumBox community. In ourdesignwe linked this social frameworkwith Rogers attributes of trialability and observability, but these attributes do not capture the feelingsof ownership and community that participants express.

Instead we conclude that while our designs focus on Rogers attributes for innovation have created invisible technology that has enabledthis macro-appropriation, they have not necessarily driven it. Instead it seems that the social efforts around the development and co-design,created a strong central group of champions, who were invested in HumBox as a concept, and in turn drew the trust and eventualparticipation of others.

6. Conclusion

Digital learning repositories are increasingly important as they provide a platform on which institutions, communities and individualscan create a visible Web presence for their teaching, with the potential of fostering innovative new professional behavior, and sharing goodpedagogical practice.

However, there are many barriers to building such a community. The most signicant of which is that participating in the rst place isitself an innovative behavior, as practitioners do not traditionally share their materials or approaches in public. Practitioners are also time-poor and risk-adversewhen it comes to adopting newapproaches or tools. Lowering barriers to participation, and creating a safe communityspace, is thus key to any successful repository.comments) enabled this, thesewere not highlighted by our participants, and again they focused on the high-level picture, seeing HumBox asa collaborative creation that tapped into existing collegiate feelings, based around common concepts and a shared philosophy.

The invisibility of technology was thus an enabler for a diffusion of ownership, from the project team itself, to core users who areenthusiastic but critical, and to effective champions who spread the message about both the site and the OER ethos.

5.2. Changing practice (micro and macro appropriation)

Across our evaluations there is evidence that HumBox is the hub for a small but focused community. Our survey and interviews revealhow this impact has translated into changing practice, both pedagogically and professionally.

The survey showed that 24% of respondents thought that HumBox had inuenced them pedagogically, with 22% actively using otherpeoples materials in their own lectures and activities. A signicant part of this was about exposing themselves to new ideas, for example inthe interviews one participant talked about being inspired by the use of Google maps:

I remember noting a resource where Google maps and other online resources were used as a research tool to explore old Bradford [http://

D.E. Millard et al. / Computers & Education 69 (2013) 287302300to explore in more depth how HumBox has impacted individual practitioners.

Daniel, J. (2013). Making sense of MOOCs: musings in a maze of myth, paradox and possibility. Journal of Interactive Media in Education, 3.David, P. A. (1990). The dynamo and the computer: an historical perspective on the modern productivity paradox. American Economic Review, 80, 355361.

Dix, A. (2003). Deconstructing experience pulling crackers apart. In M. Blythe, K. Overbeeke, A. F. Monk, & P. C. Wright (Eds.), Funology: From usability to enjoyment.

Dordrecht, the Netherlands: Kluwer.Downes, S. (2001). Learning objects: resources for distance education worldwide. The International Review of Research in Open and Distance Learning, 2(1).Eason. (1991). Ergonomic perspectives on advances in humancomputer interaction. Ergonomics, 34(6), 721741.Falzon. (1990). Cognitive ergonomics: Understanding, learning and designing humancomputer interaction. In Cognitive ergonomics, ISBN 0-12-248290-5.Fischer, G. (2009). Cultures of participation and social computing: rethinking and reinventing learning and education. In ICALT 09: Proceedings of the 2009 ninth IEEE in-

ternational conference on advanced learning technologies.Fischer, G. (2011). Understanding, fostering, and supporting cultures of participation. Interactions, 18(3).Freisen, N. (2004). Three objections to learning objects. In R. McGreal (Ed.), Online education using learning objects (pp. 5970). London: Routledge.Gaver, B., Dunne, T., & Pascenti, E. (1999). Design: cultural probes. Interactions, 6, 2129.Grudin, J., & Pruitt, J. (2002). Personas, participatory design and product development: an infrastructure for engagement. In Proceedings participatory design conference (PDC)

2002 (pp. 144161).Gruen, D., Rauch, T., Redpath, S., & Ruettinger, S. (2002). The use of stories in user experience design. International Journal of HumanComputer Interaction, 14(34), 503534.van Harmelen, M. E. (2001). Interactive system design using Oohci methods. In Object modelling and user interface design: Designing interactive systems. Addison Wesley.Hassenzahl, M., & Tractinsky, N. (2006). User experience a research agenda. Behaviour & Information Technology, 25(2), 9197.Hendrick. (1995). Future directions in macroergonomics. Ergonomics, 38(8), 16171624.Hooper, C., & Millard, D. E. (2010). Teasing apart and piecing together: towards understanding web-based interactions. In Web science 2010. Raleigh, USA.IEEE LOM. (2005). The learning object metadata standard. IEEE. Tech. rep.Kleiner. (2006). Macroergonomics: analysis and design of work systems. Applied Ergonomics, 37(1), 8189.Kresimir, F., Natasa, B.-H., & Hrvoje, J. (2010). The integration of learning object repositories and learning management systems. International Journal of Computer Science and

Information Systems, 7(3), 387407.Leech, N. L., & Onwuegbuzie, A. J. (2009). A typology of mixed methods research designs. Quality & Quantity, 43(2), 265275.Mahlke, S. (2005). Understanding users experience of interaction. In EACE European Association of Cognitive Ergonomics ACM.McAndrew, P., Santos, A., & Godwin, S. (2007). Opening up for OpenLearn: issues in providing open educational resources. In Proceedings of CAL07: Development, disruption

and debate D3. Dublin, Ireland.McAndrew, P., Santos, A., Lane, A., Godwin, S., Okada, A., Wilson, T., et al. (2009). OpenLearn research report 20062008. Milton Keynes, England: The Open University.Millard, D., Howard, Y., Gilbert, L., & Wills, G. (2009). Co-design and co-deployment methodologies for innovative m-learning systems. In Multiplatform e-learning systems and

technologies: Mobile devices for ubiquitous ICT-based education. IGI Global.Millard, D., Howard, Y., McSweeney, P., Arrebola, M., Borthwick, K., & Varella, S. (2009). Phantom tasks and invisible rubric: the challenges of remixing learning objects in the

wild. In Proceedings of the 4th European conference on technology enhanced learning.Millard, D., Howard, Y., McSweeney, P., Borthwick, K., Arrebola, M., & Watson, J. (2009). The language box: re-imagining teaching and learning repositories. In International

conference on advanced learning technologies.Our work had three objectives: to discover the scale of the engagement with the site, to measure the impact of this engagement onindividuals practice, and to assess the success of our design approach in creating a system that could be appropriated by its users.

On the scale of engagement (our rst objective) the evaluation has revealed a relatively small and focused user community of around 900registered accounts, whose participation roughly matches Nielsens theory of participation inequality (Nielsen, 2006). This also matches theanalysis work of Ochoa and Duval (2009) who showed that themajority of content is created by a few hyperactive contributors, and similarlyshows linear growth in deposits over time (assuming that we discount our pre-launch data).

On the impact of the design approach (our third objective) a key nding is that the design approach, focused on reducing complexity andincreasing compatibility, has led to technology that is invisible to users, enabling the community to focus on higher level challenges ratherthan worrying about low-level technology. In terms of understanding the impact of HumBox on behaviors and practice (our secondobjective) we see that this has resulted in reduced micro-appropriation (the use of in-the-small features like tags and collections), butincreased macro-appropriation (the use of the site in-the-large). Baraniuk (2008) found similar evidence that Connexions users saw theirsite as a vehicle to increase their impact in their discipline, and we might also characterize this as a type of macro-appropriation. In bothcases it is evidence of a diffusion of ownership that has led to changes in both professional and pedagogical practice within the community.

Above all the evaluation shows that our social framework based around the co-design activities was key in enabling both changes inbeliefs and appropriation of the technology and the greater project (objectives two and three). Although the design process was designed toenable trialability and observability, it seems that it served an even more important role as a focus of joint activity, allowing users to take amore holistic view of the project, and acting as a focus for high-level debate (such as the political discussion around OER, or the legalpracticalities of sharing). HumBox is therefore an example of Fischers approach of fostering a culture of participation by includingstakeholders as rst-class citizens (Fischer, 2009, 2011).

Our experience with HumBox has shown that techniques fromUX and co-design, that take into account social, emotional and innovationfactors, can lead to invisible software that when coupled with a strong social framework enables a community to engage with high-levelissues leading to changing practice. Web 2.0 sites that sit in the popular consciousness tend to have millions of users, and work throughsheer weight of numbers, however HumBox shows that with careful design a much smaller community can successfully use this type oftechnology to drive discussion, foster innovation and share best practice.

References

Abrahamson, E., & Rosenkopf, L. (1997). Social network effects on the extent of innovation diffusion: a computer simulation. Organization Science, 8(3), 289309.Baraniuk, R. G. (2008). Challenges and opportunities for the open education movement: a Connexions case study. In T. Iiyoshi, & M. S. V. Kumar (Eds.), Opening up education:

The collective advancement of education through open technology, open content, and open knowledge (pp. 229246). MIT.Baraniuk, R. G., Burrus, C. S., Hendricks, B. M., Henry, G. L., Hero, A. O., Johnson, D. H., et al. (2002). Connexions: DSP education for a networked world. In Acoustics, speech, and

signal processing (ICASSP), 2002 IEEE international conference on (Vol. 4; pp. IV-4144IV-4147), 1317 May 2002.Bardzell, J., & Bardzell, S. (2008). Interaction criticism: a proposal and framework for a new discipline of HCI. In CHI. Florence, Italy: ACM.Borthwick, K., Arrebola, M., Millard, D., & Howard, Y. (2009). Learning to share in the language box: a community approach to developing an open content repository for

teachers and learners. In Proceedings EUROCALL 2009.Bratina, T., Hayes, D., & Blumsack, S. (2002). Preparing teachers to use learning objects. The Technology Source. November/December.Buchenau, M., & Suri, J. F. (2000). Experience prototyping. In Proceedings of the conference on designing interactive systems processes, practices, methods, and techniques DIS 00

(pp. 424433).Busetti, E., Forcheri, P., Ierardi, M. G., & Molno, M. T. (2004). Repositories of learning objects as learning environments for teachers. In IEEE international conference on

advanced learning technologies (pp. 450454).

D.E. Millard et al. / Computers & Education 69 (2013) 287302 301Nack, F. (2003). Capturing experience a matter of contextualising events. In Proceedings of the ACM SIGMM workshop on experiential telepresence (pp. 5364). Berkeley,California.

Naja, M., & Toyoshiba, L. (2008). Two case studies of user experience design and agile development. In Agile, 2008. AGILE 08. Conference (pp. 531536).Neven, F., & Duval, E. (2002). Reusable learning objects: a survey of LOM-based repositories. In Proceedings of the tenth ACM international conference on multimedia (pp. 291

294). New York, NY: ACM.Nielsen, J. (2006). Participation inequality: Encouraging more users to contribute. Jakob Nielsens Alertbox. October 9, 2006.Norman, D. A. (July/August 2005). Human-centered design considered harmful. Interactions, 1419.Nor, R. M., & Muhlberger, R. (2010). Designing to support empathy: understanding user experience by using a model of interaction in meeting human needs. In User science

and engineering (i-USEr) (pp. 710).Ochoa, X., & Duval, E. (2009). Quantitative analysis of learning object repositories. IEEE Transactions on Learning Technologies, 2(3), 226238.Picard, R. W. (1995). Affective computing. MIT.Polsani, P. (2003). Use and abuse of reusable learning objects. Journal of Digital Information, 3(4). Article no. 164.Rogers, E. M. (1962). Diffusion of innovations. Glencoe: Free Press, ISBN 0612628434.Rosenberg, N. (1972). Factors effecting the diffusion of technology. Explorations in Economic History, 10(1), 333.Sengers, P. (2005). The engineering of experience. In M. A. Blythe, K. Overbeeke, A. F. Monk, & P. C. Wright (Eds.), Funology (pp. 1929). Norwell, MA, USA: Kluwer Academic

Publishers.Sharples, M., Jeffery, N., du Boulay, J. B. H., Teather, D., Teather, B., & du Boulay, G. H. (2002). Socio-cognitive engineering: a methodology for the design of human-centred

technology. European Journal of Operational Research, 136(2), 310323.Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage.Tzikopoulos, A., Nikos, M., & Riina, V. (2009). An overview of learning object repositories. In Selected readings on database technologies and applications (pp. 8594). IGI Global.

Web. 26 Jun. 2013.Vardi, M. Y. (2012). Will MOOCs destroy academia? Communications of the ACM, 55(11).

D.E. Millard et al. / Computers & Education 69 (2013) 287302302

The HumBox: Changing educational practice around a learning resource repository1 Introduction1.1 Research objectives1.2 HumBox1.3 Structure

2 Background2.1 Learning objects and repositories2.2 User-centered and user experience design2.3 Innovation2.4 The HumBox conceptual framework

3 Methodology3.1 Numerical analysis3.2 User survey3.3 User interviews

4 Results4.1 Numerical results4.2 User survey responses4.3 Interview responses4.3.1 Examples of changing practice4.3.2 UX factors (emotional responses to the site)4.3.3 Response to technology4.3.4 Holistics not specifics

5 Analysis5.1 Invisible technology5.2 Changing practice (micro and macro appropriation)

6 ConclusionReferences