1

Version14 23 November 2012

Reading and learning from screens versus print: a study in changing habits

Judith Stoop(1), Paulien Kreutzer (2), and Joost Kircz (3)

j.stoop@hva.nl paulien@onetwentyone.nl j.g.kircz@hva.nl

CREATE-IT Applied Research

University of Applied Sciences Amsterdam

Rhijnspoorplein 1, 1091 GC Amsterdam, The Netherlands.

(1) Also: Scherp & Zinnig. http://www.scherpenzinnig.nl

(2) Also: OneTwentyone. http://www.onetwentyone.nl/

(3) Also KRA Publishing research. www.kra.nl

And Corresponding Author

Abstract

In this paper, we report in-depth tests on reading and working with screens. In the present

transition period between information and knowledge representations on paper to

representations on screens, a series of practical as well as intrinsic issues emerged. Both are

crucial aspects for the creation of the written form and for the production of electronic

teaching material. From the practical point of view, ergonomic aspects are the most obvious,

though these aspects are in a state of turbulent flux, as about every 6 months, new and better

devices hit the market. This fact makes tests on those aspects tentative. On the intrinsic side,

we deal with issues such as the materiality of the reading substrate, the capability of making

notes, dog-ears and one’s own summaries. Furthermore, the very act of reading from different

sources, from paper, e-ink screen and LCD screen, creates a difference in the digestion of the

content. In the project Amsterdam Boekenstad (Amsterdam E-book City), we performed two

large tests comparing reading long texts from paper and from an e-ink reader. One test was

with City Council Members and one with students reading a textbook. In both cases, reading

from paper was considered superior. In two large subsequent tests, we used shorter text

materials and asked the students questions on the content of the material. In one test, we

compared print and reading from screens with the same text, which was structured differently

on paper than electronically. In the other test, we used consecutive text from print and a

mindmap representation of the teaching material. Our conclusions indicate that a one-to-one

transition from paper to screen does not work for teaching material. This in contradistinction

to the transition to e-novels. Learning with texts is not a linear activity. This means that if we

make the transition to electronic learning, not only must a series of software improvements,

such as the capability of annotation, be improved, but also the very structure of the material

needs reconsideration and modification.

1 Introduction In our research programme Amsterdam E-Boekenstad (Amsterdam E-book City), we

conducted a range of extensive tests on how professionals and students in higher education

read e-texts. In this project, we investigated the use of e-reading devices in order to

understand how authors and publishers must change their practices, traditionally based on a

paper world, towards the flexible text-representations electronic rendering of text allows.

Contrary to many informative large-scale surveys, such as Rowlands, Nicholas, Jamali &

2

Huntington (2007), Nicholas, Rowlands, Clark, Huntington, Jamali & Olló (2008), Jamali,

Nicolas & Rowlands (2009), which are based on online questionnaires and more inclined

towards measuring and forecasting the potential market, we tried to grasp the possible

differences in understanding a textbook in its various presentation forms by direct interaction

with the users. A methodical interesting study by Ackerman & Goldsmith (2011) shows that

paper is still best suited for learning. A serious issue is the rapid pace of technical

development and acceptance of electronic information and therefore an increasing

acquaintance with electronic reading by students, which makes results of many studies

(Hernon, Hopper, Leach, Saunders & Zhang, 2007; Letchumanan & Tranzi, 2011), including

our own, only tentative. This rapidly changing technological field is also the reason why we

cite mainly papers published over the last years. For a good overview for the implications of

digital text for reading in academe, we refer to Cull (2011). In a recent paper Daniel & Wood

(2013) report an elaborate study in comparing the two reading modes of the same text (print v.

screen) with about 300 students. Thought they don’t report different performance in

understanding, the act of reading was distinctly different.

The usage per se of e-books is also a serious concern for libraries, and various studies deal

with the actual use of e-books available in the library. These studies are mostly carried out by

means of questionnaires and show an increasing interest, due to, among other factors, the ease

of access (Christianson & Aucoin, 2005; Shelburne 2009; Berg, Hoffmann &, Dawson, 2010).

But those e-books are mostly just electronic copies of the paper version, often enhanced with

clickable references etc., but which leave the structure of the book as it is. So, they only

address a different way of accessing traditionally structured information and don’t yet deal

with the aspect of changing learning patterns through use of electronic materials. For a more

general discussion on the tension between technology and communication using e-readers see

Kircz (2012).

The Amsterdam E-Boekenstad project was funded by the SIA-RAAK foundation whose goal

is to foster the relationship between knowledge-seeking companies and knowledge-

generating schools. Hence, the tests were conducted in close collaboration with educational

publishers and in the first phase of the project also with an e-reader manufacturer.

The importance for publishers is not only the transition of carrier (paper to screen) but also

the ways in which content is consumed. In particular, in the case of distance learning, all

document exchange tends now to go via electronic communications; however, but does this

mean that printing at the end-user site is obsolete? This question is highly dependent on the

structure and quality of the texts dedicated to reading from screens. As our first ambition was

to understand the different reading experiences between reading from screen and reading from

paper, we were fortunate to be able to integrate a large project with city councils who also

wanted to test the pros and cons of a changeover from paper to screen.

Interestingly, when the two-year project started on January 1st 2010, the interest was still

rather lukewarm as the e-ink devices were just making inroads into the market. Despite the

harsh winds of marketing, sales, and computer aficionados, the educational publishers’ world

did not advance very fast in developing electronic teaching materials. The Amazon Kindle

was launched in autumn 2007 and became a great success for reading novels. At the other end

of the spectrum, the big scientific publishers already had their huge electronic repositories

with scientific papers in HTML and PDF. For the educational market, it was not yet clear

what the opportunities for e-readers were and to what extent this development would not only

change the business model but also, and more importantly, the way students consume

educational materials. After all, common knowledge, and also the research reported below,

that centuries of refining paper publishing created a most versatile and uniquely usable

product. In that sense, we can use the words of Hillesund (2010) that we can speak of reading

in a period of transition in the field of reading research.

3

Only after the introduction of Apple’s iPad in April 2010, did a shockwave ripple through the

publishing world. The tablet computer proved to be a portable multi-media device,

outflanking the still relatively slow read-only e-ink readers, though e-ink readers are still

superior for reading in an outdoor environment compared to backlit LCD displays. Hence, our

objectives were therefore extended from reading studies only, to wider-scope efforts to

understand educational knowledge transfer, and the relationship between pure reading as a

vehicle and the introduction of multi-media aids, such as instruction films.

Unfortunately, not much research has yet been carried out and published on the actual use of

e-reading devices under controlled conditions. Most of the published work is based on (large-

scale) questionnaires or relatively small local tests; e.g., the most recent large-scale American

report on e-reading by Rainie, Zickuhr, Purcell, Madden, & Brenner (2012) provides an in-

depth overview of the ethnicity, the age, etc., of the users as well as the usage compared to

print books, the purchasing patterns, etc., but does not break down the usages of e-books into

genres. A main extra problem is that e-reading devices are developing so fast that many

negative experiences of only a couple of years ago are now often no longer an issue at all.

Think about the loading time of a page, the speed of browsing through a text, etc. Many

ergonomical aspects are now solved.

In our research programme, we tried to avoid the obvious and concentrate on a few

fundamental aspects in the usage of e-reading devices. Our ultimate aim is to assist publishers

of educational material on how to design, write, model and shape e-books for students. In

order to do so, we executed four major tests. The first test was an attempt to measure the ease

of reading huge amounts of text from an e-ink reader. In this case, we did not use educational

texts. For this test we used the IREX1000D e-ink reader

(http://en.wikipedia.org/wiki/Digital_Reader_1000 ) which had a large screen of 26 cm (10.2

inches) in diagonal, which made it an excellent device for reading PDF files in A4 format (the

hegemonic paper size in Europe). In order to guarantee a heavy reading load as well as an

enthusiastic and disciplined corpus of readers, we collaborated with nine city councils in and

around Amsterdam. In these tests with 45 people over a period of two months, all the

documents council members received for their meetings were uploaded on the Irex1000

readers.

In conjunction with this test and apart from two device usability tests, not reported here (1),

with students of the University of Applied Sciences in Amsterdam, whose qualitative results

are interesting as a check list for designers, we performed three major tests comparing reading

from a screen and reading from paper, in each test working with a different educational

publisher. In the first test with students, we could use a book that suited the study programme

perfectly. In the two subsequent tests, the emphasis was more on learning from e-text per se,

without prior knowledge of the subject. The choice for the specific study material (stimuli)

was also dependent on what the publisher wanted to test and had available.

We started with 80 students, who, for an examination, had to read a book of Van Duuren

Media Publishers ( http://www.vanduurenmedia.nl ), from the Irex1000D, a laptop and on

paper, respectively.

In the next test together with the educational publisher SDU (http://www.sdu.nl ), with 196

students, we compared the scores made in tests of comprehension of a text, which was the

same in content, on paper, and on an LCD screen, but differently arranged.

In the final test with the educational publisher Noordhoff ( http://www.noordhoffuitgevers.nl )

with 173 students, we compared the results between learning from a paper source, from

1 All full reports of the entire project are available in Dutch online at http://eboekenstad.nl

4

learning with the aid of a website with complementary materials, and from learning with the

use of a mind map, with all the add-ons integrated in a digital file on a computer.

We analysed the results of these last two studies statistically. In addition, we had extensive

group discussions with the participating students in order to interpret the results.

We first briefly discuss below the issues dealing with on-screen reading, and subsequently we

report on all four tests in detail.

1.1 Measuring usage Research on reading from electronic devices is not straightforward. We must consider the

following aspects: reading as such, navigation between documents, navigation within

documents and technical aspects.

1- Reading as such has related ergonomical aspects such as: zooming, layout, sharpness, font

design, etc., and physical aspects such as: wearing of the eyes, and the feel and smell of a

book. These aspects have everything to do with the materiality of the device. On this issue,

we see currently important neurological research, comprehensively presented by, e.g., Wolf

(2007) and Dehaene (2010). This research is closely related to the way we write and how texts

are made and its materiality as discussed by Haas (1996). Mangen (2008, 2013) presents

research about nonlinear reading. For educational texts, the issue becomes even more

important though here we are dealing with an ever-stronger interaction between plain text,

graphs, drawings and photos, captions, videos and increasingly interactive elements. The

reader not only consumes the material, but must be able to internalize the content and also be

able to reproduce it. Hillesund (2010) concludes that online immersion is very different from

the hermeneutics of traditional reading. This is exactly what we try to understand, as making

new - digitally born - learning materials must start from this observation. Noyes & Garland

(2006) concluded from an elaborate questionnaire filled out by 274 participants that books are

viewed more favourably than computers, primarily due to the physical and practical aspects of

the two media. In their final conclusion, they state that as books and computers are different

tools and both are equally useful, we need to build upon this in the learning situation. In other

words, what content can be provided through computers and when is a print medium

superior? This is what we tried to find out in our studies.

2- Navigational aspects linking between pages such as: flipping through the pages, skipping

pages, book marking and dog-ears, searching for words, paragraphs, sections or chapters, etc.,

is manifestly evident. It is important to stress that reading an educational book, or as it is in

our first test, political documents, is different than reading a novel. In the latter case, we

“thumb” through the pages, eager to reach the conclusions. But in our tests, consecutive page

turning is not an issue at all.

As said above, the fast development of e-reading devices makes it almost impossible for some

features to use data older than a year. The rapid development of – entertainment-driven –

devices makes it difficult to make a comparative study of the pros and cons of the various

devices. For a recent comparative overview of devices, see Gibson & Forbes Gibbs (2012).

Our choice for the Irex1000D ( http://support.irexnet.com ) was based on the fact that it had

the largest screen available and hence full pages could be displayed equal in size to the paper

book page. The Irex Technologies Company was also a partner in the project, but had to file

bankruptcy in June 2010 due to the slow take-off of it sales in the US. As an unfortunate

result we did not have the opportunity to test the - much faster - successor (IREX DR800SG).

This means that some negative reactions on the use of e-ink, in particular issues dealing with

the speed of page loading and browsing, could not be re-evaluated.

5

3- Navigational aspects on the sentence level, such as: references in the text to and from other

text elements, illustrations, endnotes and footnotes, etc. This aspect deals with hyperlinks

within a text, which is an essential feature of electronic publications. Nevertheless, given the

materials we tested, this aspect did not become a research object in the present study. For this

discussion, see Kircz (1998), Kircz & Harmsze (2000) and Kircz & Den Boef (2013).

4- Ergonomical aspects such as: portability of the device, battery lifetime, the need for

ambient light in the case of e-ink readers versus backlit screens in the case of LCD screens.

Recent usability evaluations are given by Kang, Wang & Lin (2009), with an overview by

Cull (2011).

This aspect is also a standard issue in many studies on the actual use and acceptance of e-

reading devices, such as the large-scale surveys by the London CIBER group: Rowlands,

Nicholas, Jamali & Huntington (2007), Nicholas, Rowlands, Clark, Huntington, Jamali &

Olló (2008), Jamali, Nicolas & Rowlands (2009), already mentioned.

The above list is just a beginning, as presently most e-books are representations of paper

versions. The basic format of a book is still the page, which is a strictly paper-oriented

remnant of the past. The size of a paper page is typically a historical result of the human

reading habit. The eye does not like too long sentences, and also too short lines are usually

judged as unpleasant to read. In newspaper typography, we have clear rules for the column

width and as it turns out, left and right justification of the line of type reads more easily than

ragged right line-endings. Moen (2000) names the following items: legibility is determined by

at least 8 factors: (1) type design, (2) type size, (3) line width, (4) word spacing and letter

spacing, (5) leading, or line spacing, (6) form, (7) contrast, and (8) reproduction quality.

In the case of electronic devices, the page as a unit for information as well as a graphical

container of content becomes obsolete: page-based standards such as PDF are like polaroids

of oil paintings; they try to mimic the old, but do not share their intrinsic features. There is a

good reason for doing so, as almost half a millennium of reading experiences with print

resulted in many proven concepts of typography and layout.

The biggest change until now is ‘reflowable’ texts as we have in most e-book formats. Here,

the page is often, but not always, kept as it was on paper, but enabling various physical

formats of screens to represent the text. The big design challenge is not how to manipulate the

sentences, but how to keep the unity of text, illustrations, tables, white lines, etc., which are

essential for comprehension of most art, educational and scientific books as well as many

poetic works.

The truism of this observation can be seen daily in the huge increase of sales of fiction and

non-fiction books in an e-format like Epub, Mobipocket, etc. reported by all internet

bookshops, which does not demand a fixed relationship between running text and

illustrations.

Electronic schoolbooks are different and similarly with scientific books: the option of screen

rotation, or switch between portrait and landscape modes, plays havoc with the page layout in

cases where lay-out is essential, as indeed it is in many fields.

Below we will discuss our four research studies. Firstly we report two qualitative studies

dealing with the issue of sustained reading from screens. Subsequently we report two studies

under controlled conditions in which we also tried to address the differences in text structure

between paper and electronic texts. The reports are followed by a conclusion and

recommendations for e-study books. The first test was conducted with professional readers,

6

the others with students from our school, and institutions of higher education for

professionals.

2 First test: Sustained reading from a screen by city council members 2.1 Introduction

Reading novels on an e-reader is becoming a normal practice, but what about other types of

documents, where layout and structure are important? In this test, we investigated the ease of

use and the constraints of professional reading of large piles of documents on e-readers. Our

partners in this sub-project were the companies Notubiz (http://notubiz.nl/) and Docwolves

(http://docwolves.nl/ ). These collaborating companies in document management systems

work for a great variety of, mainly public, organisations such as city councils. Originating

from a minutes-taking company, they developed an elaborate document management system

including the electronic distribution of all documents, covering the order of business of

meetings, the archiving, as well as of now, the video registration and indexing of city council

meetings. Their research in the distribution of structured documents for reading from the

screen dovetails with the needs and questions of educational publishers.

2.2 Research objective

The general research quest was to what extent the central distribution of all kinds of

documents covering meetings from the Council Information System to individual council

members could be organised using a paper-free e-reader environment? An important aspect

for our partners, the registrars and commercial partners, is the routing of versions and the

dynamics of mailing lists, as in the process of policy-making, these lists change when

confidentiality aspects change. We do not discuss this further in the present paper because it

pertains more to information management systems than to reading content. But even if all

logistics were solved, the pertinent question of document consumption by reading remains the

main issue. The preparations started in autumn 2009, whilst the actual tests were executed

between August 2010 and March 2011.

Our central quest was: how do council members and members of the registry use and

appreciate the use of an e-reader in their work, and what points for improvement could be

identified? Note that we allowed participants to download e-books of any kind onto their

reader in order to get acquainted with e-readers, but restricted our questions and evaluation

discussions to reading from council meeting minutes.

2.3 Method

With the active participation of the registrars and their clerks, we approached various city

councils in the Amsterdam area to interest council members in tests in which they would

receive all documents on an e-reader during a period of about two months. In most cases, they

also received the traditional pile of paper documents; a few enthusiastic participants insisted

on working electronically only. In most cases, we used the Irex1000D, as described above.

The unique, crucial features of the device are the great legibility of text and the screen size,

which allow a good reading of the standard A4 format.

During council meetings, constant reference is made to particular sentences, clauses and other

items in documents. And after deliberations, the various texts must be voted on. For that

reason, the fixed page layout is essential.

Before the test, a questionnaire was sent to all 241 council members of the participating

councils to find out how the actual document handling was performed. After the tests, we held

evaluation discussions to find out what the experiences were. We opted not for use of a

second questionnaire, as it was not a quantitative test but a qualitative one to try and find out

7

real usage and experiences. In this report, we focus on the qualitative conclusions as they

inform us about the possibilities of using e-readers in professional reading. As, during the

tests, many council members became accustomed to reading from screens and now use

notebooks, tablets, e-readers, etc., our interest was focussed on what can be improved if we

change from paper to screens. As mentioned in the introduction, the field is moving rapidly

and many ergonomical and logistic aspects improve by the day. So, we omit those comments

that are irrelevant for the appreciation of present-day e-readers.

2.4 Sample

We performed tests in 9 cities and boroughs in the wider Amsterdam area. The participating

council members and registrars received their documents via Notubiz’ infrastructure by

logging into a dedicated server. Depending on the enthusiasm, 5 to 10 people per council

worked with e-readers within a minimum of 9 weeks. This period was chosen because it

allowed for a minimum of 2 commission meetings and 2 full council meetings.

In 2 cities, Amstelveen and Zaanstad, next to the Irex1000D, we also used the iPad 1, which

had just hit the market. In the case of Amstelveen, the test, which was more a first

exploration, was not fully the same as with the others councils, but we incorporated the results

as expressed in the evaluation discussions in our conclusions. In the case of Zandvoort,

unfortunately no clear reportable results emerged and they are left out in the results. In table

1, an overview of the cities and participants who completed the test is given.

Table 1: Overview of the cities and participants of this study

City Inhabitants

rounded off

Council

members

Participants Percentage

female

Youngest

age (years)

Oldest

age

(years)

Almere 190,700 39 4 50% 33 71

Amstelveen 81,800 37 2 No data

Amsterdam-

Borough

Centre

82,700 29 4 0% 25 61

Amsterdam

Borough –

West

13,000 29 6 16,6% 26 50

Edam-

Volendam

28,600 21 5 60% 19 57

Waterland 17,100 17 4 50% 38 61

Zaanstad 146,900 39 11 45% 31 64

Zandvoort 16,600 17

Zeevang 6,350 13 4 75% 51 68

2.5 Results:

2.5.1 The document deluge

At the start of the test, a questionnaire about the actual use of paper documents was sent to all

council members of the participating cities, totalling 241 people, of which 90 returned it on

paper and 22 digitally, (total of n=112 or 46%). On average, council members receive more

than 100 pages of text per meeting. In meetings where complicated dossiers are discussed,

such as budgets and zoning plans, the amount of paper can go up to several hundred pages per

meeting. Often, (parts of) the documents – in an earlier version – are being distributed to

council commission meetings before final versions reach the full council meeting. For council

8

members, which in all cases are only part-time politicians, the information overload is large

and storing and handling is complicated.

In 80% of the cases, council members received the documents on paper as well as

electronically, per e-mail or as a file to be downloaded from the council web page; 13%

received a paper version only, and 7% an electronic version only. Given the number of pages

involved, this means that the logistics and paper consumption are considerable. This was one

of the reasons for the registrars to join this project.

In the case of updates - e.g., new versions - we saw an increase in electronic dissemination to

only 17%. Interestingly 12% of the respondents did not compare updates of the documents

with the originals, whilst 15% said that they compare new documents, line by line, with

previous versions.

On average, council members spend 17 hours per week on this work, most of the time in

meetings. When a member is the spokesperson for a subject, 98% of them will carefully read

the documents. However, if a member is not the spokesperson, only 10% of the members

fully read the documents. As said above, all documentation for this, as well as documentation

for the following tests, are available in Dutch on our website.

2.5.2 Paper document use As with educational reading material, the documents are information-carrying tools and

writing on the documents is part of the process of digestion of the content. More than 30% of

the respondents underlined sentences, scribbled in the margin, used markers etc. About 7%

used post-it stickers or look-alike aids to mark text. 6% makes dog-ears to retrieve the

relevant pages and 16% used separate paper sheets to note all comments, often on the first or

last page of the document in question.

It goes without saying that in a digital-only environment, this memorizing and commenting

behaviour has to be transcended into electronic aids. Only 5% of our respondents throw all

paper documents away immediately after the meeting, independently of whether or not the

issue at stake will return on the agenda. 12% retain those documents that deal with still-

pending subjects. 60% retain documents they consider important as such. A solid 16% retain

all paper documents. In the case of electronic versions, which remain on the online archive of

the council, 37% of the members indicated to retain everything on their own computer.

2.5.3 Electronic document usage As asked in the evaluation discussions, the participants rated the importance of the various

functionalities typical for the Irex e-reader, targeted to their use as council member (and not,

e.g., by using the device for reading downloaded books). We discuss the answers below

starting with the most important feature.

1) Readability. Participants were most pleased with the legibility of the text and the ease of

reading. Unfortunately, if the letter size was enlarged (e.g., in order to read without

spectacles), the overview of the page was lost. This is typically the dilemma between the

order of the pages that must be the same for all users, and reflowable text as in e-novels where

font size can be changed without consequences, but where the coordination with other users

gets lost. In that case, two or more readers cannot refer to the same page anymore, since fixed

pages are no longer to be found.

2) Search function. Searching in the documents hosted in the e-reader is possible. Though a

strong demand came to the fore that online searching for related information, not only in the

council document archives, but also on internet pages or services, was imperative, here we see

that the step from paper to an electronic substrate immediately induces demands that belong

to electronic storage as such. In the paper world, a simple index was considered sufficient, but

9

as soon as the step to electronic is taken, all available electronic features are considered

relevant and are demanded by all.

So, going over to an electronic document, all electronic capabilities have to be incorporated.

Whilst our prime concern was the ease and quality of reading, these aspects became essential.

Furthermore, the need for systematic indexing of documents by subject came to the fore, as

free-text searching is not sufficient. At present and also as result of these conclusions, Notubiz

has all public information of the cities they work with in an online searchable archive

(http://www.politiekarchief.nl/ ).

3) Text editing. Text editing and making notes on the text were perfectly possible with a

special stylus. Unfortunately, the speed of the stylus was too slow. This is a typical hardware

and software problem on how overlays or changes in documents by the reader can be

implemented and stored. This technical issue is still in full development in all e-reading

device developments.

4, 5, 6) Browsing speed, battery lifetime, and memory. These are typical issues that

continuously improve performance. These comments made are largely obsolete now, as the

newest e-readers and tablets perform much better within every 6 months.

7) Screen size. Here we touch on the important issue of full-page presentations. Present- day

tablets have about the same size as the Irex1000D and are well suited for full-page viewing.

That it was not rated as the most important feature might be induced by the fact that the pages

were actually readable in full, and no tests with smaller screens were performed.

8) Weight. This was considered no problem as our test people were used to stacks of paper.

9) Private use. Interestingly, those testers using the ink-reader did not see a serious usage for

private activities. We have to take into account that the explosion of e-novels was still to

come. However, the few users, in Zaandam and Amstelveen, who used the iPad 1, did

indicate that it would be good for usages other than only for reading official documents.

2.5.4 Conclusions of test one One of the interesting aspects of this test was the logistics and coherence of the various

documents streams. After all, a set of documents pertaining to various subjects is discussed

and often voted on at meetings. On the one hand, we see the issues of version control,

including auxiliary documents such as motions and amendments to policy papers that are

often tied to a particular version of a document. Here, we have a subject-related collection. On

the other hand, we also have collections that overarch many subjects, such as all documents in

preparation for budget and control meetings. On top of that, many policy papers are deeply

related to others, e.g., the budget for school building renovations with plans for an auto-free

street, the role and place of the public library and many aspects of social welfare. In that

sense, we see an environment with a rich multi-dimensional structure, fit for structured

hypertext. In the test, the documents for one meeting were integrated as one large file with

internal hyperlinks. Nevertheless, people preferred to receive the documents piecemeal, in

particular because most council members want to read documents and prepare interventions

and possible amendments and motions stretched out over the week. Reading from an e-reader

was certainly an advance as they could take, so they say: “the whole pile of documents”

wherever they went. The e-reader also mimics the private ‘archive’ and enables one to look

back at earlier related discussions, versions and drafts.

For all those reasons, we concluded that in order to go over to a digital-only environment, we

have to start with a proper document management system and clear logistics that enable

reading from multiple reading devices, including print on paper, as well as clear indexing of

related documents. This last point is particularly tedious, as, after new elections, the division

of fields in commissions changes. A public space might be one with housing and squares and

in a next phase in the same cluster as traffic and the environment. Thus, the labels to

10

documents in a parliamentary period, and even the way they are phrased (Greens and Parks,

can be renamed in Environmental Planning) do not always guarantee a consistent wholeness

with the contents discussed.

For the registrars involved, it became crystal clear that the way documents are written and

prepared must change. Traditionally, the documents are built up in a scheme that starts with a

whole review of how this issue came into the discussion and what already has been said and

done in the particular case. On paper people can simply skip those pages, in order to go to

those parts that demand a vote. Browsing on a reader or scrolling on a screen is a different

thing. The materiality of paper allows for quick flipping to and from pages with the traditional

aids as dog-ears and sticky notes. In an electronic environment, it turned out to be much more

cumbersome. This induces a discussion on the changing practice of the writing order and

structure of a document.

It is important to note that these problems are obviously similar with current electronic

educational environments, where courses can change, and often are renamed, whilst keeping

large chunks of information unchanged. E-reading is not only a matter of reading, but is

intrinsically deeply tied with the structure of the document and subdocument management.

The pile of documents with a green marker for all documents dealing in some way with parks,

greens and trees stowed away in the hall closet, cannot be transferred to an e-reader without a

total redesign of the documents at issue.

3 Second test: paper versus laptop versus e-ink e-reader

3.1 Introduction

In our second study, 81 students from our ‘Marketing, Media, Publishing’ programme

participated. We used the marketing textbook ‘Digitale Marketing & Communicatie’ (Digital

Marketing & Communication’) by Schuurmans (2008), which is part of the curriculum and

required reading for the subject ‘Online Marketing’. This study took place in the period from

December 2009 to February 2010.30 students studied from the printed book. Another 28

students received the material as a PDF file and studied the material on their own laptops. The

last group of 23 students received the same PDF version, but used the E-ink Irex1000D e-

reader, mentioned above. The important issue here is that we deal with the same page layout

in all cases. Interesting studies of Chong, Lim & Ling (2008, 2009) report the student

preferences for the pure navigation and layout aspects of PDF versions of books. These

recommendations are important in order to convert a page to a web environment, but do not

deal with the structure of the content.

3.2 Research objective

Our central research question was defined as: “How do students, who enrol on a course on the

subject ‘Online Marketing’, experience and appreciate studying from their prescribed book by

using a PDF-file on an Irex1000D, the same PDF-file on a laptop, and the print book version,

respectively?”

3.3 Method

Since - especially at that time - not much was known about the use and appreciation of e-

books in higher education, this research project was exploratory and had a qualitative set-up.

The primary goal was to gain insight and understanding. Before, during and after the test

group discussions with the students were held. In the first round of discussions, before the

experiment took place, 30 students participated, while 66 students participated in the

discussions during and after the tests. Students were asked to keep a logbook. The discussions

11

during and after the experiment were analysed together. The total duration of the experiment

was 12 weeks.

Just as like in our first study, reading the material provided was needed in order to function in

a real-life setting, rather than laboratory tests as the subsequent tests are. Participants were

motivated to give it their best try, since it affected their own functioning in school. The broad

discussion meetings on the teaching material and the reading experience showed sufficient

insight to abstain from statistical analysis, which would not provide deeper knowledge about

the issues at stake.

3.4 Results 3.4.1 Test group Irex users

The students placed in the ‘Irex group’ were very enthusiastic about it at the beginning, but

when the actual experiment started and time passed, their enthusiasm diminished quickly.

The logbooks they kept told us that initially they used their Irex quite often, not so much to

study from, but to show other interested people, who had never seen an e-reader before at that

time. They tried and tested all functions and quickly built a negative image of the e-reader.

All in all, these students concluded that the PDF-file on an Irex offered little to no added

value compared to a paper book. This conclusion concurs with the conclusions of the first test.

During the test period, the students were asked to study only from the version they received at

the beginning of the test. The most striking result was that almost none of the ‘e-reader group’

students ended up studying from the Irex; they put the Irex aside.

They felt using the Irex for the e-book version was an obstacle to their learning process. The

biggest obstacle was that since the Irex was relatively slow, both in starting up as well in

processing and ‘turning pages’, they couldn’t study properly with it. Studying people do not

read a book from cover to cover, but jump from page to page, and chapter to chapter. They go

to and fro through texts and compare pages, pictures and tables. The e-reader was too slow for

this type of reading. Another problem was that the e-version only shows one page at the time.

It does not permit fingers or pieces of paper between pages, though bookmarks are available.

This is a general problem of e-books. It looks more a scroll than a book. Furthermore, in the

e-version, students could not make notes easily. The Irex does allow making notes with a

special stylus, but writing legibly demanded serious training and patience. A remarkable

insight was that the ‘e-reader group’ postponed studying for their exam a longer time than

seen in any other groups. The expectations they held beforehand about studying with an e-

reader couldn’t be met. Apart from technological disadvantages, the design of the e-reader

wasn’t appealing either. They could not believe that it was in black and white only, and found

it too large to take with them in their bags, which is remarkable as the paper book was not

much smaller but heavier and less vulnerable. The e-reader appeared not to be as cool a

gadget as they thought it would be. E-ink readers are well suited for continuous reading, but

lack the functions of a laptop. The perception of the students was that a novel device must

incorporate all functions, a thing nobody expects from a book. Hence, the change from paper

to e-ink was considered old-fashioned and not useful. As it turned out, the students from the

‘e-reader group’ either bought the paper book, used the e-book on a computer or laptop, or

found ways to print the e-book.

3.4.2 Test group laptop users

Furthermore, the students using a PDF-file on their own laptops experimented with the

possibilities of the e-book. They soon found out that there was not much they could do with

the file. In their logbooks and in the group discussions, they complained that they could not

mark, save changes, or copy and paste. Since a large part of this group were used to making

their own summaries on the computer, it would have been very handy if they could have

12

manipulate the document. Apart from technological constraints, another big disadvantage

concerns the lack of easy reading. Since this text was a book of about 200 pages, many people

complained about weary eyes. The e-book was protected, so they could not ‘copy and paste’

parts in order to make a personal summary, nor make notes in the document. Most

importantly, the laptop they needed for reading their book, turned out to be a very distracting

medium. Pop-ups from e-mail, Facebook and other social media often spoiled their

concentration. It was a disappointment that this ‘e-book’ was nothing more than a scanned

version of the paper book, whereas technology - in principle - would make it possible to add

more dynamic functions such as audio or video.

The biggest advantage was that now their book was ‘mobile’. They could consult it from any

location, since it was downloadable from the web. In practice they always had it with them as

they carried their laptops wherever they went. Another advantage was the search function.

The clickable index was also appreciated.

Interestingly, half of the ‘laptop group’ students cracked the code of the protected e-book and

printed it on paper. Learning such a large text from screen did not work for them.

The people that did learn from the screen either could not crack the code or considered buying

the paper book as too expensive. As with the Irex group, the laptop group did not consider

learning from screen an improvement, it didn’t provide any added value for the students who

used them. Indeed it hindered them in their studies.

3.4.3 Test group paper book users Finally the paper group: at first they thought they would miss out on an interesting

experience, being in the ‘boring’ group, but afterwards, they had nothing to complain about

when they learned about the negative experiences of their fellow students. From their

logbooks, it becomes clear that of all students, this group of students spent relatively the

longest time studying. But since many students from groups 1 and 2 switched back to paper

during the experiment, data are hard to compare. Students studying the print book told us they

studied as they usually did, they didn’t meet obstacles, they read great parts of the prescribed

texts and they made summaries. The ease of use of the print book is great, since they are used

to it. Their concentration during their studies was high, since they were not easily distracted

by other functionalities of the ‘device’. The only disadvantage they could think of would be

the weight of the book. But since they were also used to carrying books, this was not

considered a real disadvantage.

3.4.4 Conclusions of test two

One conclusion we can draw from this experiment is that our students were willing to try

something new. Functionality turned out to be the key criterion for continued use or not. The

functionality of both the e-book itself (an ‘ordinary’ PDF-file that was overprotected and

therefore couldn’t be manipulated) and the e-ink reader were very disappointing for these

students and hindered them in their studies. Obviously, reading for study is something quite

different from reading a novel. While studying, one wants to be active with a text, as was the

case with our test with council members: that means marking, making notes, copying and

pasting. Furthermore one doesn’t start on page 1 to end at the last page, but one needs to go

to and fro through a text. This is in line with the conclusion of Ackerman and Goldsmith

(2011, p29) that the decisions to print digitally presented material before study might be

viewed as a meta-metacognitive control decision that transfers the study materials to the more

subjectively reliable context of paper learning. So the navigation technology must be easy and

speed must be high. Students of groups 1 and 2 experienced little to no added value; on the

contrary; they struggled with the immature technology. The ‘e-book’ offered nothing more

13

than the print book in a scanned version. Students from groups 1 and 2 experienced a lack of

overview, due to a lot of necessary scrolling in the PDF-file on the laptop and due to technical

imperfections, in particular page-turning speed of the Irex. This is also the conclusion of

Woody et al. (2010, p945), in a test with 91 students half of whom used an e-book version.

They also conclude, as we do in this paper, that the design of an e-book may need to differ

from that of a textbook to make a more constructive user experience. Furthermore, Brunet,

Bates, Gallo III & Strother (2011) conclude in a 9- month study with dental students that the

electronic bookshelf leads to mixed opinions.

So, at the time of this research project, neither e-reader technology nor e-books were

sufficiently advanced to add significant value when reading for study purposes.

Apart from the required faster and more advanced technology, these students also expected

the e-book content and design to be innovative. An e-book should be more than a simple

digital version of a print book. It should also make use of opportunities like adding sound and

video and also linking it to internet. The Irex was not perceived as a ‘cool gadget’ (whereas

the iPhone is). The design reminded them of the first black-and-white televisions, which

interestingly they do not know from their own experience. They talked about it as ‘that thing’,

which shows that they did not find it especially appealing.

4 Third test: learning from paper versus computer screen

4.1 Introduction

After our more qualitative tests on sustained reading from screens, we turned to more

controlled tests in which we confronted students with new material, not part of the

curriculum, but still within their intellectual reach. The main, practical, reason for this was

the availability of teaching material from our publisher partners.

A group of 196 students from the University of Applied Sciences of Amsterdam took part in

our third experiment. Here, we used a text on internal communications within companies

(Van Riel, 2010). This text was not part of the curriculum, but had relevance within the

broader reach of the study programme. An important aspect here and in the subsequent test is

that we dealt with a single aspect of a larger treatise. As it became clear from all our

discussions, students have the tendency to read only those parts of a book the teacher

prescribes, therewith limiting their self-education to the formal minimum of examination

requirements. Hence, in breaking up learning lines into comprehensible chunks introduces the

danger of a limitation of the context of the teaching material.

We created two research groups. Half of the students worked with a paper version, whilst the

other half used the computer. The paper version comprised a couple of paragraphs from the

book, a separate dictionary and a separate list of rehearsal questions derived from the material

at the end, all put together in a paper reader.

The other group studied exactly the same text, though presented on a series of seven

consecutive web pages. In this version, the text was restructured to fit onto a computer screen.

A mouse-over enabled the dictionary (so, a small pop-up window containing the lemma

becomes visible if the mouse hovers over a word), while test questions were situated together

with the relevant parts of the text.

4.2 Research objective

In this test, our purpose was to test the following hypothesis: “When study material is offered

in an interactive form, the learning results of the user are higher than when the same material

is offered in print.” A secondary goal was to understand why one method would score better

in learning results than the other method. Furthermore, we wanted to discuss the future of

study material with these students in order to be able to make recommendations to our

14

collaborating publishers. In this test, we worked together with SDU Publishers, who provided

us with both the print version as well as the electronic version of the text. This experiment

took place in June 2011.

4.3 Method

The first part of this study entails a quantitative experiment to detect possible differences in

‘learning results’ between the two groups. The groups read the texts and could check their

understanding via rehearsal questions. These results were measured in a separate knowledge

test, containing 24 questions that all participating students had to complete immediately

following 25 minutes of study. The experiment took place in a controlled situation. The

students were randomly assigned to the two experimental conditions:

1. Experimental group (n = 100) – using the interactive version on a PC.

2. Control group (n = 96) – using the print version.

In order to gain further understanding on why one method would score better than the other,

we held group discussions with 31 students. These were mixed discussions that took place

immediately following completion of the knowledge test was, and in a different room of that

where the tests were carried out. Apart from gaining insight into how the students had

experienced the way of learning, we also took this opportunity to brainstorm with them about

the future of ‘study material’.

4.4 Sample

196 students from the School of Design and Communication participated. Participation was

voluntary, but participants received €10 as a teaser. Table 2 shows how they were randomly

assigned to each one of the groups. Please note that RMC (Advertising, Marketing and

Communication), MMP (Media Marketing Publishing), IM (information and media), RMP

(Editing Media Production), NM (News and Media) and IAM (Interactive Media) are all

specific discipline branches of study directions within the school.

Table2: Study compared to experimental condition

What is your discipline?

Total How did you study?

From print From pc

% % %

First year Communications 24.0 24.0 24.0

RMC 18.9 18.8 19.0

MMP 7.1 5.2 9.0

IM 6.6 7.3 6.0

RMP 6.1 8.3 4.0

NM 7.1 6.3 8.

IAM 18.9 18.8 19.0

other study 11.2 11.5 11.0

N 196 96 100

4.5 Results 4.5.1 Quantitative results

In 6 of the 24 test questions, the ‘print group’ scored better, though statistically speaking, this

was not significant. On the other hand, in the remaining 18 cases, the ‘PC group’ scored better

of which 6 had a statistical significance of 90% or higher. Chi-square analysis was done in

order to determine whether or not there were significant relationships between learning result

15

per test question and experimental condition. The result of these 6 questions that showed a

significant relationship are presented in Table 3.

Table 3: Overview of test questions showing significant relationships

Question % correct in print

group (control

group)

% correct in computer

group (experimental

group)

χ2 % of significance

1 87.5% 97% 6.25 98%

4 90.6% 97% 3.46 90%

5 78.1% 89% 4.24 95%

8 85.4% 98% 10.34 99%

15 88.5% 95% 2.73 90%

20 61.5% 73% 2.97 90%

When one looks at the structure of these 6 questions, no striking similarities can be found. In

other words, it is not a specific type of question that made the computer group score

significantly better. This hypothesis can be confirmed for 6 out of the 24 test questions, with a

level of significance of at least 90%.

4.5.2 Qualitative: why the ‘pc group’ scored better

Group discussions were held with 31 students. In these mixed discussions, students explained

how they had experienced studying in their respectively different ways. This helped us to

better interpret the results of the quantitative part. The conclusions can be summarized as

follows.

Students have problems with long consecutive texts and prefer reading comprehensive

chunks, immediately followed by exercises and questions, as was the case with the ‘computer

group’. Contrary to the second study discussed above, where students had to read a whole

book, this study material consisted of ‘only’ 7 web pages with auxiliary texts. These pages

fitted the screen, so no scrolling was needed. None of the students complained about weary

eyes or a lack of overview.

Students study actively, which means that they make notes and summarize read material in

their own words. This turns out to be very difficult with present-day electronic equipment.

Scrap paper was available for all participating students. Although it was not measured how

many of them used scrap paper exactly, it was clear from the amount of used paper that most

of them did.

The rehearsal questions by themselves were appreciated by both groups, but used differently.

The ‘computer group’ used them intensively, while the ‘print group’ hardly used them at all.

Rehearsal questions help to determine what the main issues of the text are. The fact that in the

electronic version, the questions are related to the text, contrary to the print version where the

questions were placed at the end of the text, acted as stimulation to the groups to answer all

the questions and to do so immediately. Students of the ‘computer group’ first read the

question, and then the text, in order to get a better understanding of the text. Students of the

‘print group’ did not attempt to search out the questions while reading (although the

introduction indicated that there were questions offered at the very end of the text), and only

consulted them after reading, and only when they had time left. If they had more time, they

would have liked to have a better look at them. So, these students read and used the material

in the order that it was presented to them.

For the ‘computer group’ the dictionary was enabled by a mouse fly-over. This acted as a

stimulation for them to actively use the dictionary while reading the text, whereas the ‘print

group’ only consulted the dictionary after reading, at the end of the text where it was situated,

and only if they had time left.

16

We, of course, cannot prove a correlation between actively using the rehearsal questions and

the dictionary on the one hand, and having better scores on the test on the other hand (because

there might have been other influences), but this more likely explains the higher scores of the

‘computer group’. Students were better able to digest the knowledge if the primary text, the

dictionary and the rehearsal questions, were intertwined. Of course, this tells us something

about the modes of learning of the students in our applied educational environment. This

might be different in a research university setting and hence suggests the need for

comparative tests.

4.5.3 Qualitative: the future of study material

Our group discussions with the students ended with a brainstorm session about the future of

study material. So this part of our study might help to promote an understanding of what the

actual future will look like as seen from the present. It indeed helps us to understand what

these students think, here and now, and what, is realistic and/or useful to them.

An interesting result is that these students would like to get rid of the term ‘book’. Why

should study material be presented (mainly) as books? With new technologies and interactive

possibilities, one might wonder whether a book is still the most effective way to transfer

knowledge. Almost all of the students, more-or-less, considered a kind of website or computer

program that would replace books in the future. It would be something that would consist of

multiple layers. The upper layer would be a sort of summary, and if one wants to know more

about a topic, one can click to enter deeper levels of understanding.

Written text, as a way to transfer knowledge, will always remain the basis, according to these

students. This is mainly due to the external pacing of written text. The texts should be short

and manageable. Scrolling should become something of the past. Having a good overview is

very important. Texts should be accompanied by video, audio, interactive tests and games. In

this way, studying could become more of a ‘total experience’.

All students we spoke to during our different studies told us they like to study in a more active

fashion. That means ‘doing’ something with the material. In the easiest way, that can be

explained as marking, making notes and making summaries, but with new technologies

‘doing something with the material’ could mean much more, such as: doing tests, interactive

games, rehearsal questions, etc.

Furthermore, social media could be integrated within the material, so people can work

together on topics and/or ask questions to each other, and have discussions. And one last

advantage of ‘e-study material’ is that the author can update the content.

One of the most interesting results of our brainstorms sessions, however, is that almost all of

these students got really excited and enthusiastic when thinking about future possibilities.

They thought study could become a really pleasant and exciting activity this way.

4.5.4 Conclusion of test three

Our hypothesis that the general learning results would be higher in the group that studied in

the interactive form can be confirmed for 6 out of the 24 test questions. Probable reasons for

this result are that students who used the interactive form were more likely to use the

dictionary and use the rehearsal questions as part of their learning process, whereas the

students in the ‘print group’ only consulted these two ‘extras’ at the end and only if they had

time left over. For this last group, these features were not perceived as an integrated part of

the learning material. Therefore, we can tentatively conclude that the order and way in which

learning material is presented influence the degree in which it is actually used and consulted.

Extra features, such as a dictionary or rehearsal questions, should be firmly tied in to the

relevant parts of the text.

17

5 Fourth test: interactive mind map versus printed text with additional

website

5.1 Introduction

A one-to-one translation of a traditional printed study book into an electronic book, e.g., in

PDF form, does not provide any significant added value, as our second study illustrated. Only

if the electronic book has some additional interactive features, such as a dictionary that pops

up via a mouse-over, and where rehearsal questions are related to the relevant text parts, does

the learning result appear to become affected positively, as our third study indicates. Now

what happens if we take this electronic form one step further? That is what we did in our

fourth study. Here we collaborated with the educational publisher Noordhoff, who provided

us with the test material. For this test, we used ten pages from the book ‘Geowijzer’ by Peters

& Westerveen (2011). These are texts about Dutch landscapes. This subject has no direct

relation to the curriculum of these students, but provides a nice mix of text and video and is

intellectually on a par with the level of our students.

Again we tested differences in learning results. In this case, roughly half of the participants

studied from an interactive mind map (see: http://nyjmolen.home.xs4all.nl/mind map.htm ),

and the other half studied from a print reader supported by a website (see:

http://nyjmolen.home.xs4all.nl/geowijzer.htm) offering video material, summaries, rehearsal

questions and a dictionary. We chose for this combination, as currently, more and more study

books offer a supporting website with additional features students can consult. It is important

to mention, that in this test, both experimental conditions offered literally the same text and

the same additional features. But both forms of presentation differed completely.

5.2 Research objective

The hypothesis we wanted to test was: “When study material is offered in an interactive mind

map, the learning result is greater than when the same material is offered in print supported by

an additional website”. Even more than with our previous study (test 3), the relevant material

was tied together in the interactive mind map (i.e., video’s, dictionary, etc. were located right

at the relevant part of text), whereas the students in the control group (print with website) had

to look up the additional features on a computer that was in front of them and preset on the

right web page. A secondary objective was to gain insight into why one method would score

better than the other.

5.3 Method

The first part of this study entailed a quantitative experiment to determine possible differences

in ‘learning results’ between the two groups. This was measured by a knowledge test of 24

questions that all participating students (n = 173) had to answer directly after 30 minutes of

study. The experiment took place in a controlled situation. The students participated

voluntarily, but received a €10 reward, and were randomly assigned to the two experimental

conditions.

The first experimental group of 82 people used the interactive mind map; the control group of

91 people used the print version with supporting website.

In order to gain a further understanding as to why one method would score better than the

other, we held group discussions with 25 students. These were mixed discussions that took

place immediately following the knowledge test in a separate room.

5.4 Sample

18

172 of all 173 participating students studied at our School of Design and Communications of

the University of Applied Sciences in Amsterdam.

5.5 Results 5.5.1 Quantitative

The group that studied from print with supporting website scored better on 8 out of the 24 test

questions. The mind map group on the other hand, scored better on the remaining 16 test

questions. The first impression is therefore that the mind map group had a higher learning

result. However, when we restrict ourselves to statistically significant differences only, it

appears that the print group scored better. On the 8 test questions that this group scored more

highly on, in 3 cases it was significant, hence meaningful. Conversely, the mind map group

scored significantly more highly on only one question.

Table 4 provides an overview of the questions where significant differences between the 2

groups were found. The calculated χ2 should be greater than 2,706 for a significant difference

with a reliability of at least 90%. The higher the calculated χ2 value, the more meaningful the

observed difference and the higher the reliability rate are.

Table 4: Overview of test questions with significant differences between the groups

Question % correct in print

group (control

group)

% correct in mind

map group

(experimental

group)

χ2 Percentage of

significance

1 83.5 93.9 4.55 95%

5 65.9 48.8 5.20 95%

16 92.3 84.1 2.81 90%

20 93.4 82.7 4.77 95%

It appears that the non-significant differences were often not even "almost significant", and

where they are, they equal out for both groups. We must conclude that the results are far from

unequivocal. Nor can we conclude that one study method works better than the other.

Increasing the sample size would probably not lead to clearer results, since there is no

evidence that this would be in favour of one of the test groups. For the learning results, it did

not matter much whether participants were studying from print with supporting website, or

from the interactive mind map. Both forms had advantages and disadvantages. This is further

discussed in the group discussions. So, we have to reject our hypothesis that the interactive

mind map gives better study results in this context.

5.5.2 Other interesting quantitative findings

Apart from answering the 24 test knowledge questions, the participants also answered some

other questions about their behaviour. These responses are analysed below.

Firstly, we wanted to know how many video clips the students had viewed. In the interactive

mind map, 4 video clips (about the texts) could be found. The clips were related to the

relevant text part (and therefore set apart from each other). For the other group, the 4 video

clips were presented side-by-side on the supporting website. The % of students that watched

all four fragments is significantly higher in the print group (42.9%), compared to the mind

map group (22%). But interestingly, the % of students that did not watch any video at all is

also slightly higher in the paper group (30.8%), compared to the mind map group (26.8%).

The students in the print group were more inclined to view all videos, once they had decided

to watch one, clearly because the 4 videos were presented side by side. In the mind map

group, we see more variation in the number of videos viewed. This is presumably due to the

fact that in the mind map, the video clips were spread over the map. Students studying from

19

the mind map watched 2 or 3 clips more often compared to the students from the other group.

Thus, the study method has an influence on the number of videos watched, but it is not such

that, with one method, more videos were viewed in absolute terms, than with the other

method. χ2 analysis confirmed a significant association between the study form and the

number of videos viewed. χ2 is 19,4, indicating a reliability of at least 99%. If videos are

offered together, the student is more inclined to watch all the videos. However, if videos are

presented separately, but related to the relevant text parts, students are more inclined to watch

the video immediately after or before reading the relevant text parts. Table 5 shows the

percentages.

Table 5: Amount of watched videos compared to study method

Did you watch the available

video clips?

Total 1. How did you study?

Print, with additional

website.

Interactive mind

map.

% % %

yes, I watched 4 videos. 32.9 42.9 22.0

yes, I watched 3 videos. 15.0 6.6 24.4

yes, I watched 2 videos. 12.7 7.7 18.3

yes, I watched 1 video. 10.4 12.1 8.5

no, I didn’t watch any

videos. 28.9 30.8 26.8

N 173 91 82

The participants were allowed, if they wished, to use scrap paper during the study. They had

to hand their notes in before the knowledge test started. In our previous study (study 3), we

observed that many students used scrap paper. Therefore, in this test, we decided to measure

this use and see whether study method had an influence on the use of scrap paper. However,

there appeared to be no significant relation between these variables (χ2 is 0.82). The results

are shown in Table 6. We also analysed qualitatively the way the different groups used scrap

paper, but there were no remarkable differences in how they used it, see also point (5) below.

Table 6: Use of scrap paper compared to study method

Did you use scrap paper while

studying?

Total How did you just study?

Paper, with additional

website.

Interactive mind

map.

% % %

no, I

didn’t. 22.5 25.3 19.5

yes, I did. 77.5 74.7 80.5

N 173 91 82

Finally, the students were asked which group they themselves thought would score better on

this test. This led to an interesting result. The students believe that students from their own

group will score better. In the print group 79.1% shared this opinion. The students of the mind

map group were less outspoken. It should be noted that the students answered that question

without actually knowing how the other study method worked. Hence, their belief is based on

a prejudice. The relationship between study method and the expectation of which of the

groups would score better, is highly significant (χ2 is 27.33, so the level of significance is over

99%). See Table 7.

20

Table 7: Expectation of test result compared to study method

With this experiment, half of the students

studied from print with additional website,

the other half studied from an interactive

mind map. Which group scores better on the

knowledge test (that you also just made)?

Total How did you just study?

Print, with extras on

the computer.

From the

Mind map

on the

computer.

% % %

The print group' 60.7 79.1 40.2

The 'mind map group' 39.3 20.9 59.8

N 173 91 82

5.5.3 Qualitative: the use of the interactive mind map

Students first had to get accustomed to the mind map, as for them it was a new way of

representing teaching materials. In the discussions, the following items emerged. The biggest,

perceived, advantage of the mind map is that by itself, it represents a kind of folded

‘summary’. In particular, since these students had only 30 minutes to study the material, such

a summary was appreciated. For larger amounts of learning material, and if they would be

allowed to study for longer than 30 minutes, which is the case in regular learning situations,

the mind map is considered questionable. A folded ‘summary’ felt too constrained. They

suggested a nervousness of missing important things.

On the other hand, a disadvantage of the interactive mind map is the perceived lack of

overview. The computer screen becomes an obstacle between the students and the mind map.

The unfolded mind map is very large and a large fraction does not fit the screen. This means

that the screen window has to be manipulated over the map. In particular, this becomes

problematic as at the start it’s unclear how large the unfolded mind map actually is. Thus, the

students could not easily estimate how much material they had to study and accordingly how

to spend their available time. Although the mind map in the test was perceived as a summary,

many students still felt the need to make their own summaries on scrap paper. Writing things

down is part of the learning process. This brings us to an interesting conclusion that clearly

flipping through book pages does provide an easy understanding of the total volume of the

material, while moving a screen window over a large two-dimensional map does not.

5.5.4 Qualitative: the use of the print reader with additional website

Almost everybody in our evaluation ‘print’ group also consulted the additional website. This

mimics the situation that is now common practice, that study books offer an additional

website with extra material. There was no threshold in our setup to do so, since the computer

was placed right in front of the students and pre-set on the specific webpage. Interestingly, the

students told us they hardly ever consult the extra sites of their study books. They consider it

too much trouble to swap from their paper book to the computer, find the specific website and

use necessary logins codes. This hurdle was removed in our test.

Again (as in our previous studies), we can conclude that students use/read/consult study

material when it is presented to them in a convenient and integrated way. Packing together

different technologies does not improve the learning experience. For the educational

publishers, our students suggested that the additional material on the websites must be

advertised at the relevant place in the text of the book. They also should make it very clear

what can be found on the site, and why it should be accessed. An additional website can be

very helpful, these students think. It could help them fulfil the need to study actively, by

offering extra explanations, summaries, definition trainers, video material or even educational

21

games. The conclusion of these discussions is again that various media have to become

integrated and mutually referenced in a coherent and systematic way. It must be clear from

the learning process why, when, and how a swap between screen and paper is a favourable

activity in the learning process.

5.5.5 Qualitative: the use of scrap paper

In all previous experiments, we encountered the pressing issue of reader annotations. In this

study, we followed Haas (1996, in particular ch.4), by looking at the scrap paper used. White

paper sheets were handed out to all participants and collected afterwards. Most students did

indeed use scrap paper, see Table 6.

Interestingly, apart from scribbles, almost all paper was used to make notes, overviews and

summaries, and in many cases just quotes. It clearly proves the old observation, everybody

also knows from her/his own experience that writing up something is a most important mental

aid for the memorization and internalization of knowledge. This is also stressed by Grafton

(1999) when he discusses the new active way of reading, and a clear break with the scholastic

past, by humanists in the late fifteenth century, when the book began to be used for

annotations and for abstraction and reformulation of the content by the reader.

Summarizing is a filtering process. By summarizing, the reader emphasizes her own priorities

and decides on which topics is no further attention needed.

More research is needed on the question of to what extent making notes on separate sheets of

paper or making a summery by typing on a computer, makes a difference. Though, on a

computer ‘cut & paste’ is an easy way of summarization, it misses the active reformulation of

the content via writing. It goes without saying that on the computer, students are more able to

create order, because that is what they are after. However, the didactic question remains to

what extent rewriting helps in internalizing the content. Scribbles are found frequently; their

role in the learning process also demands further investigations.

5.4.6 Conclusion of test four

There are no clear differences in the learning results of the group that studied from the

interactive mind map, compared to the group that studied from print with the help of an

additional website. We have to reject the hypothesis, for this situation, that the mind map

enhances the results in an unequivocal way.

The mind map was perceived as a summary, which was considered as most useful when one

has only 30 minutes study time. For larger texts and more elaborated topics, an interactive

mind map would probably be less practical. The main negative aspect of the mind map was

that students felt a lack of overview, since they could not predict how large the map would

become; the unfolded mind map didn’t fit the screen size by a long way. The screen actually

stood in their way as a hindrance to a comprehension of the full richness of the mind map.

The students in the print group did visit the additional website and found that useful.

‘Normally’ though, they told us they are not so eager to consult additional websites that come

with study books. It is perceived as too much trouble and offering too little benefit. Therefore,

additional sites should be clearly integrated with the text and each other, with high quality and

relevance as well as being easily accessible. They should really offer added value, and when

they do, these students would very much welcome them.

For the way people study, it matters in what order study material is presented. If videos are

offered together with the other material, then students are more inclined to watch all the

videos. However, if videos are presented separately, connected to the relevant text parts,

students are more inclined to watch the video immediately after or before reading the relevant

text parts.

22

So, in order to let students make optimal use of available material, material should be located

at relevant places and not be ‘hidden away’ at the end or put away on an additional website

with no ‘natural’ accessibility. An important issue reported by Zacharis (2011), is the

relationship between learning style and material used. Further investigations along those lines

demand longitudinal studies, which were unfortunately beyond our reach, though, Emerson &

Mackay (2011) report better results for students who learned the lessons from print rather than

online; no pertinent reasons could be identified.

6 Summary, general conclusions and recommendations

In our programme, Amsterdam E-book City, we have tried to understand how reading from a

digital screen could influence the quality of learning and how the learning material might be

reconstructed in order to increase efficiency and effectively. With this in mind, we started

with the issue of reading per se, using an e-ink reader known for its good jitter free screen

quality. In the first test we collaborated with nine City Councils in the Amsterdam area.

Council members are dedicated readers with a professionally high reading load. The question

was to what extent the document flow and the handling of documents could be improved by

electronic means. During this test period, with third generation e-ink readers, the iPad was

introduced. In a way, this interfered with the original quest, as now more functions, such as

communication and web searching became available. However, it sharpened our research

goals. Reading from an e-ink reader turned out to be appreciated on essentially the same

principles as why people love novels on an e-reader. It is portable, clearly readable and a

reader can host a great many documents. Apart from the more technical aspects of documents

delivery in tune with the meetings scheme of the councils, using e-readers was considered an

advantage compared to print, as long as we deal with the reading experience as such.

The real problem turned out to be the capabilities of browsing, annotating and referencing.

Despite electronic aids, dog ears and sticky notes are superior in re-finding a pertinent

paragraph. Note taking could be done on the device but a piece of paper did better.

Furthermore, the fixed format of an A4 print with hand written notes is easier to comprehend

than electronic comments. We have to take into account that we used PDF files, in where, at

that time, notes could not be integrated.

A more intrinsic aspect that emerged was the very structure of the documents. Parliamentary

documents have a more-or-less fixed structure. People are used to skipping easily parts where

they do not expect to find those particular aspects they are looking for. Electronic documents

can have a much more modular structure, enabling different readers to have different reading

paths. Furthermore, switching to and fro between related pages and having various related

documents “open” is an important demand. In discussions with the Council Registrars, this

was considered an important issue in an attempt to reduce the workload of council-members

while keeping the integrity of the completeness of the documents. These discussions

dovetailed with the ideas developed earlier on the modularisation of academic texts (Kircz

1998, Kircz & Harmsze 2000, Kircz & Den Boef 2012, and references therein).

Interestingly and not unexpectedly, the results of this reading test also merged with the

subsequent studies. Partly simultaneously, together with our partner Van Duuren Media, we

tested 90 students divided into equally large groups. These students read a book that was

preparatory for an examination. One group got the book as a PDF-file on an e-reader, one as

PDF on their laptop and one read the book from print. Here, reading as such, also turned out

not to be the main issue. Again, browsing and annotating turned out to be essential.

Interestingly, opposed to the council members who considered the integration of electronic

23

documents with online searching an important novel asset, our – professional – students

considered the pressure of online communications very distractive. Print won in all categories.

Interestingly, in our subsequent studies, students rated the integration of social media in the

study material of great value in order to exchange findings and problems, although this could

not be substantiated. In a way, one can phrase this as a tension between the discipline of the

book as an independent object and the desire to share knowledge with peers, though within

the context of the learning material.

Subsequently, we oriented our research more on the comparison of the same text in different

presentations, as acquiring knowledge is a complex process surpassing the reading of a text.

In these tests, both with almost 200 students, we asked the students to study a text for about

25 minutes and subsequently fill out a knowledge test. With a selection of the students, round-

table discussions were held. Our test students showed a great interest in a precise definition of

the material, exemplified in the better results of the group who used well-defined chunks of

text on the computer as compared to those who used the full consecutive print version. In the

case of a mind map presentation the lack overview was a serious point.

Students commented that teachers often only prescribe parts of books and therefore inferred

that only part of the material is worthwhile reading for an examination. Here we encounter a

serious issue, as it exemplified the tension between the conscious build-up of the material by

the expert author of a book, on the one hand, and on the other hand the course goals as

defined by the local teacher. This goal is mostly defined by the demands for passing an exam

and not necessarily for full comprehension of a field. This fact together with the conception of

many students that you do not have to read whole books to pass the exam, we are immediately

confronted with the challenge as to how to write and edit study material that stimulates the

students to read and view, in the case of videos rather than the bare minimum. A deep

knowledge of the various interactions between a basic text, auxiliary material, and related

texts is needed and demands further longitudinal research. Note that our school is geared to

educate professionals and not academic researchers, which makes such research difficult to

undertake.

In all our tests, the use of scrap paper turned out to be a universal aid. The writing of

keywords, short sentences or mini-abstracts helps to internalise the content. The same is true

for scribbling in the margins, underlining and highlighting with a marker. In the various

discussion groups, students stressed the tactility of the book. Having a document in your hand,

and keeping it close is also a way for isolating the learning material from the outside world.

Working on a laptop was distracting because of the many other applications “at your

fingertips”.

Two main conclusions can be drawn here.

1) In the transition from paper study materials to electronic materials, a full capability for

marking, scribbling notes, etc., is imperative. This does not mean that people cannot become

accustomed to all kinds of note-making applications in document-handling software.

However, there remains a big difference between writing, typing, and ‘cut & past’. In our

opinion, this conclusion is a hard demand for the development of educational learning

materials.

2) The second conclusion is that authors have to be aware that reflective questions and

exercises have to be designed differently when an e-reader or computer is used. The relation

between questions, exercises and the running text must be integrated. Turning pages is

something completely different from scrolling down screen text. The place and use of all

auxiliary texts via mouse-overs, or hyperlinking, demands intense research in order to allow a

24

new screen-based order in which the integrity and integration of the main text and other texts

is guaranteed and where switching between the main text and auxiliary texts becomes fluent.

As mentioned above, our investigations were carried out in the middle of a stream of

changing technology. The screen technology and software developments are still changing

continuously. In that way, the other bank of the stream is still ill-defined and receding.

Whilst in our first test, the speed of turning pages on an e-ink reader was indicated as an

obstacle, the latest versions of e-readers are many times faster. In addition, the screens of

tablets improve year by year, slowly closing the distance in readability between e-ink screens

and LCD screens.

In this paper, we also report the first tests in providing teaching material in different forms,

using print text versus differently structured electronic text as well as a mind map and an

auxiliary website.

Though we can be sure that the hardware and software in question will considerably improve

over the next years, we can conclude from these early tests that, in the presentation of

educational material, form and content are firmly connected. The more data or facts, the easier

an electronic model will work. The more reasoning, examples and digressions are needed, the

more clearly will written text with integrated multimedia components and test questions be

needed. This all indicates that electronic text-books and other electronic learning materials are

much more than a collection of snippets of text linked to a database of pictures, videos and

audio files. The new way of reading and learning, using electronic devices, allows for fast and

comprehensive delivery of materials but also induces new ways of composing, structuring and

mutually relating the various presentations of the underlying knowledge and instructions.

Further research is needed in longitudinal studies to find out to what degree habituation plays

a role, as well as in-depth tests with authors and students on how the print structure of a text-

book can be transposed to the more flexible electronic future.

7 Acknowledgements

We thank our colleagues Gerlof Donga, Jacob Molenaar, and Jos Vrolijk for their continuing

support during the course of the project. Alexa Brinkman participated as research student in

our first test. Her persistency was of great value. We thank Keith Jones for his many helpful

corrections. We also thank Rens Groeneveld of Notubiz/Docwolves, Bob van Duuren of

VanDuurenMedia, Marieke Gierveld of Noordhoff andTannet Remmelts of SDU for their

kind collaboration and interactions. The project received a grant of the SIA-RAAK foundation

under number 2009-14-3H, which is gratefully acknowledged.

25

8 References

Ackerman, R. & Goldsmith, M. (2011). Metacognitive regulation on text learning: on screen

versus on paper. Journal of Experimental Psychology: Applied. 17(1), 18-32.

Berg, S.A., Hoffmann, K. & Dawson, D. (2010). Not on the same page: Undergraduates’ s

Information Retrieval in Electronic and Print Books. The Journal of Academic Librarianship.

36(6), 518-525.

Brunet, D. P., Bates, M.L., Gallo III, J.R,, & Strother, E. A..(2011). Incoming Dental

Students’expectations and Acceptance of an Electronic Texbook program. Journal of Dental

Education. 75(5), 646-652.

Chong, P. F., Lim, Y-P. & Ling, S.W. (2008). E-book design preferences: A case study.

ITSim 2008. International Symposium on Information Technology, 2008,.1(1-8), 26-28 Aug.

2008

doi: 10.1109/ITSIM.2008.4631538

http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4631538&isnumber=4631524.

Assessed 7 Nov.2012.

Chong, P.F., Lim, Y-P. & Ling, S.W. (2009). On the design preferences for ebooks. IEEE

Technical review. 26( 3), 213-222.

Christianson, M. & Aucoin, M. (2005) Electronic or print books: Which are used. Library

Collections Acquisitions, & technical Services. 29, 71-81.

Cull, B. W. (2011). Reading revolutions: Online digital text and implications for reading in

academe First Monday, 16(6).

http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/rt/printerFriendly/3340/2985.

Assessed 7 Nov. 2012.

Daniel, D., B. & Woody, W., D. (2013). E-textbooks at what costs? Performance and use of

electronic v. print texts. Computers & Education 62(1), 18-23.

Dehaene, S. (2010). Reading in the brain. The new science of how we read. London: Penguin

Books.

Emerson, L. & MacKay, B.. (2011). A comparison between paper-based and online learning

in higher education. British Journal of Educational Technology. 42(5), 727-735.

Gibson, C. & Gibbs, F. (2012). An evaluation of second-generation ebook readers The

Electronic Library, 29(3), 303-319.

Grafton, A. (1999). The Humanist as Reader. In G. Cavallo & R. Chartier: A History of

Reading in the West (pp. 179-212). London: Polity.

Haas, C. (1996). Writing Technology. Studies on the materiality for literacy. Mahwah:

Lawrence Erlbaum Ass. Inc. Publishers. Reprinted by NY: Routledge 2009.

26

Jamali, H.R., Nicolas, D. & Rowlands, I. (2009). Scholarly e-books: the view of 16.000

academics: results from the JISC National E-Book Observatory. Aslib proceedings, 61(1) 33-

47.

Hernon, P., Hopper, R., Leach, M.R., Saunders, L.L. & Zhang, J.. (2007). E-book use by

students: Undergraduates in Economics, Literature, and Nursing. The journal of academic

librarianship. 13(1) 3-13.

Hillesund, T. (2010). Digital reading spaces: How expert readers handle books, the web and

electronic paper. First Monday, 15(4-5).

http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2762/2504

Assessed 19 November 2012.

Kang, Y-Y, Wang, M-J. J. & Lin, R. (2009). Usability evaluation of E-books. Display 30, 49-

52

Kircz, J. G. (1998). Modularity: the next form of scientific information presentation? Journal

of Documentation. 54(2) 210-235.

http://www.kra.nl/Website/Artikelen/Jdoc98.htm

Kircz, J.G. & Harmsze, F.A.P. (2000). Modular scenarios in the electronic age. In: P. van der

Vet en P. de Bra (eds.) CS-Report 00-20. Proceedings Conferentie Informatiewetenschap

2000. De Doelen Utrecht (sic), 5 April 2000.31-43. :

http://www.kra.nl/Website/Artikelen/mod2k.html

Kircz, J. (2012). E-readers are for Reading: An Informal Consideration of Reading's Next

Steps. Scholarly and Research Communication 3(2).

http://www.src-online.ca/index.php/src/article/view/74/84. Accessed 19 November 2012.

Kircz, J. & Den Boef, A. H. (2011). Writing differently in the digital era. Hamlet in:. J.G.

Kircz and A. van der Weel (eds.) Hyperborg. Proceedings The Unbound Book Conference,

Amsterdam 2011.

In production 2013.

A final draft version is available at:

http://www.kra.nl/Website/Artikelen/Schoon-%20KirczDenBoef-29-8-2012.pdf

Letchumanan, M & and Tarmzi, R.A. (2011). E-book utilization among mathematics students

of Universiti Putra Malaysia (UPM). Library Hi Tech 29(1), 109-121

Mangen, A. (2008). Hypertext fiction reading: haptics and immersion. Journal of Research in

reading, 41(4), 433-453.

Mangen, A. (2013). The digitisation of narrative reading: Theoretical considerations and

empirical evidence. In: J.G. Kircz and A. van der Weel (eds.) Proceedings The Unbound

Book Conference, Amsterdam 2011.

In production 2013

27

Moen, D. (2000). Newspaper Layout & Design: A Team Approach by Daryl R. Moen Iowa

State University Press. Fourth edition.

http://web.missouri.edu/~moend/design/books/nld5.htm

Assessed 19 November 2012

Nicholas, D., Rowlands, I., Clark, D., Huntington, P., Jamali, H.R. & Olló C.(2008) UK

scholarly e-book usage: a landmark survey. Aslib Proceedings Vol 60(4) 311-334.

Noyes, J. & Garland, K. (2006). Explaining students’ attitudes toward books and computers.

Computers in Human Behavior 22(3), 351-363

Peters, A. & Westerveen, F. (2010). Geowijzer. Groningen. Noordhoff Uitgevers B.V.

Rainie, L. Zickuhr, K., Purcell, K., Madden, M., Brenner, J. (2012). The rise of E-reading.

Pew Reserach Center’s Internet & American Life Project.

http://libraries.pewinternet.org/2012/04/04/the-rise-of-e-reading/

Riel, C.B. M. van. (2010). Identiteit en Imago. Den Haag, SDU uitgevers.

Rowlands, I., Nicholas, D., Jamali, H.R. & Huntington, P. (2007). What do faculty and

students really think about e-books. Aslib Proceedings. 59 (6), 489-511.

Schuurmans, U. (2008). Handboek Digitale Marketing en Communicatie. Culemborg, Van

Duuren Media.

Shelburne. W. A. (2009) E-book usage in an academic library: User attitudes and behaviors.

Library Collections, Acquisitions, & technical Services. 33(2-3), 59-72

Wolf, M. (2007). Proust and the Squid. The story and science of the reading Brain. NY.

Harper Collins Publishers.

Woody, W. D., Daniel, D.B. & Baker, C. A. (2010). E-books or textbooks: Students prefer

textbooks. Computers & Education 55(3), 945-948

Zacharis, N. Z. (2011). The effect of learning style on preference for web-based courses and

learning outcomes. British Journal of Educational Technology. 42(5) 790-800.