Young children accessing multimodal texts: A case study1

Dr Polyxeni Manoli & Professor Maria Papadopoulou

University of Thessaly, Department of Early Childhood Education, Volos, Greece

Abstract

The study aimed to explore whether young children, who did not have formal reading

skills, could speculate the linguistic message depicted in contemporary multimodal

texts based on information provided in various modes (such as, colour or typography).

The contribution of each mode to the meaning-making process was investigated as

well. A further aim of the study was to probe into students’ answers in relation to their

age. The sample of the study consisted of 45 young children; 26 of the participants

were five to six years old and 19 were four to five years old. The basic tool used in the

research was a matching exercise derived from a picture book intended for young

children. Semi-structured individual interviews were used for the data collection,

which were tape-recorded and, later on, were transcribed and processed. The results

of the study indicated that the majority of the children were able to use the

typographic and colour cues along with separate letters they already knew to guess the

linguistic content of short multimodal texts. It was also revealed that colour and

typography played a critical role in helping students match the linguistic elements

with the visual ones. Regarding the age variable, it was found that the group of the

older children seemed to perform better than the younger ones but the difference was

not found to be statistically significant. At the same time, the results highlighted the

fact that children at this age tend to rely more on visual cues when approaching

written texts. The study pointed out the need for educators to extend the limits of the

semantic field of literacy beyond language texts in order to allow for multimodal

texts, as nowadays, students, even from an early age, are exposed to an increasing

dominance of multimodal texts -both print and digital texts- that involve a complex

interplay of linguistic elements, visual images, graphics, and design elements.

1 Manoli, P. & Papadopoulou, M. (2016). Young children accessing multimodal texts: A

case study. In the Proceedings of the 12th World Congress of Semiotics http://www.iass-

ais.org/proceedings2014/view_lesson.php?id=178

Keywords: Semiotic Modes, Visual Literacy, Multimodal Texts, Preschoolers

1. Theoretical issues

Till recently literacy education in the Western societies has focused solely on

language, ignoring all other communicational modes of meaning-making (Kress and

Van Leeuwen 2006). However, in the last decades of the 20th

century, the

technological development along with the wider changes in the socio-economic and

cultural field imposed radical changes in the representational and communicational

landscape that led to the need to redefine and broaden the limits of literacy in order to

fulfill the current and future communication needs of all citizens (New London Group

1996). Contemporary multimodal texts require a new definition of literacy as new

learning needs have arisen (Unsworth 2001). People, especially youths, are

continually exposed to multimodal texts either print or digital, such as websites, video

games, picture books, texts, magazines, advertisements, and graphic novels that

include a complex interplay of written texts, visual images, graphics, and design

elements (Kress et al. 2001; Kress and Leeuwen 2006; Unsworth 2001). As

Fleckenstein (2002) asserts readers encounter more multimodal texts in their everyday

life than print-based texts. In this context, meaning is derived from ways that are

increasingly multimodal (Cope and Kalantzis 2000). According to Kalantzis and Cope

(2012: 2), “we need to supplement traditional reading and writing skills with

multimodal communications”. Multimodality refers to the active and dynamic

interrelationship among the different semiotic modes of meaning which individuals

can draw on during interaction with various texts to derive meaning, though one mode

can prevail over the others (Baldry and Thibault 2006; New London Group 1996). In

addition, research in the field of multimodality has highlighted new ways of

conveying meaning from texts emphasizing the fact that becoming an efficient

reader/viewer of multimodal texts is much more multitasking than just reading the

words of a text. At the same time, multimodality, in particular the visual mode, can

help young readers who have not developed formal reading skills access a text

(Arizpe and Styles 2003; Pantaleo 2005; Papadopoulou 2001a; Yannicopoulou 2004).

Images, colour or even typography can provide cues revealing the information in a

text. Indeed, research has shown that children can even be more efficient readers of

pictures than adults (Kiefer 1995; Meek 1988).

Among other texts, picture books for children have always relied on the visual

mode to make meaning. Although there is quite a lot of research on how children read

images in picture books (Arizpe and Styles 2003; Kiefer 1995; Styles and Arizpe

2001; Walsh 2003), only a small body of research regards picture books as

multimodal texts and provides tools for the understanding of the contribution of the

various modes, such as, typography or colour, to the meaning-making process

(Papadopoulou, Manoli, and Zifkou 2014; Serafini and Clauzen 2012; Walker and

Linda 2003; Walsh 2003). In this context, acknowledging the contribution of the

various semiotic modes to the meaning-making process, the purpose of the present

study was to explore whether young children (4-6 year-old preschoolers) could

speculate the linguistic message of multimodal texts included in a picture book based

on information provided in various modes. Additionally, the study investigated the

contribution of each mode to the meaning-making process as well as the relation

between children’s answers and their age.

2. Method

2.1. Research questions

Allowing for the theoretical framework discussed above and the purpose of

the present study, the following research questions were addressed:

� Could young children, who did not have formal reading skills, identify

meaning in contemporary multimodal texts derived from picture books

based on information provided in various semiotic modes (such as,

colour or typography).

� Which of the semiotic modes contributed most to the meaning-making

process?

� Was there an association between children’s correct answers and their

age?

2.2. Participants

Forty-five (45) children aged four (4) to six (6) years old participated in the

study. Twenty-six (26) of the participants were five to six years old and 19 were

four to five years old. The sample of the study came from four state nursery schools

in the city of Volos, in central Greece, while one nursery school located in the suburbs

of the city of Volos. The choice of the sample relied on the following criteria: the

children did not face learning difficulties or any other mental disorder; they had not

developed formal reading skills. The two first criteria were judged based on their

teachers’ perceptions, while the development of formal reading skills was assessed

through a short test administered prior to the study. All participants had Greek as their

first language (L1).

2.3. Research instrument

The basic tool used in the research was a constructed matching exercise drawn

from a picture book, which was entitled Me + you = together. The art of being a

grandmother (Marantidou 2010). The book was written in Greek but the specific

extract was translated into English by the researchers. The specific extract was chosen

for its variety of semiotic modes that provide visual salience by means of

conventional or more expressive typography and colour. Initially, the page consisted

of images and short phrases beneath each image explaining and extending what was

symbolically depicted in each image. However, the page was modified for research

purposes; the phrases were separated from the relevant images to construct a matching

exercise. The first part of the phrases was written in normal black typefaces, while the

second part was written using semiotic modes which diverged from the normal black

typefaces.

The first image depicted a birthday cake with a candle, while the

corresponding phrase was ‘the land of birthday’. The special element of the phrase

was that the letter ‘i’ (‘ι’ in Greek) of the word birthday (γενεθλίων in Greek) was

written in a larger typeface than the rest letters and the accent was yellow bearing a

resemblance to the flame of the candle.

‘the land of birthday’

The second image depicted pairs of feet dancing and hands moving around

while the relevant phrase was ‘the land of dance’. The distinctive feature of the phrase

was that the last part of the sentence was written three times on three different lines,

which diverged from the regular line spacing and the horizontal dimension of the rest

of the phrase to remind readers of the concept of dancing.

‘the land of dance’

The third image depicted a Christmas tree, while the corresponding phrase was

‘the land of Christmas’. What made the phrase diverge from the normal black

typefont was the use of colour; namely, half of the letters of the word ‘Christmas’

(Χριστούγεννα in Greek) were in red, while the rest were in green to resemble the

colours used to depict the Christmas tree in the image.

‘the land of Christmas’

The fourth image showed the sun, while the related phrase was ‘the land of

light’. The distinctive feature of the phrase was the use of colour, which was gradually

changing; namely, the word ‘light’ [φωτός in Greek] was written using the shades of

yellow, such as red or orange, to help readers associate the phrase with the sun.

‘the land of light’

The last image depicted a piano, while the corresponding phrase was ‘the land

of music’. The distinctive features of the phrase were that the letter ‘o’ of the Greek

word ‘µουσικής’ (music in English) was replaced by a note to remind readers of the

concepts of the piano and music; moreover, the specific word diverged from the

regular line spacing and the horizontal dimension of the rest phrase.

‘the land of music’

2.4. Data collection and coding

Semi-structured individual interviews were used for the data collection, which

were carried out in Greek, the participants’ mother tongue. The interviews were

conducted at the different nursery schools and lasted for 20 minutes approximately.

The interviews were tape-recorded and, later on, were transcribed verbatim and

processed to have objective record and reanalyze data after the interviews being

conducted (Nunan 1992). The researchers coded the data independently until they had

reached 90% agreement (inter-rater reliability) on the coding of the participants’

answers. In cases in which disagreement on the coding scheme occurred, the

researchers met to discuss possible differences in order to arrive at a high level of

consistency concerning the number of students’ correct answers and the type of

semiotic modes identified by preschoolers as the reason which helped them answer

correctly (Charmaz 2000; Patton 1990). The thorough data management and analytic

procedures, such as the verbatim transcription and accurate records of the interviews,

contributed to the validation of the research findings. The research belonged to a

broader survey on preschoolers’ ability to derive meaning from multimodal texts,

which was conducted in spring of 2011 and lasted for three months (Papadopoulou et

al. 2014).

3. Results

The data of the study were analyzed using the Statistical Package for

Social Sciences (SPSS) version 17.0. In accordance with the aims of this study,

descriptive statistics as well as the statistical analysis of one-way Anova were

computed. The level of significance was set at .05. To examine children’s correct

answers, descriptive statistics, particularly frequencies, were computed. It was found

that the majority of children were able to match the linguistic message with the

correct image drawing on the semiotic modes used to convey meaning. Table 1

presents the frequencies and percentages of students’ correct guessing for each image:

Table 1

Frequency of students’ correct answers per image

IMAGES FREQ. PERCENT

1 30 67%

2 25 56%

3 29 65%

4 38 85%

5 27 60%

Total 45 100

In a next step, it was attempted to investigate the contribution of the various

semiotic modes to the meaning-making process. Based on Table 2 below, it was

shown that primarily colour and then typography played a critical role in helping

students match the linguistic elements with the images, since most of the children

seemed to have mainly relied on these two modes in order to justify their answers.

Table 2

Children’s reported justifications

IMAGES REASONS PERCENT

1 Typography

Colour

Letters

Other

55%

37%

-

7%

2 Typography

Colour

Letters

Other

No answer

28%

20%

21%

21%

7%

3 Typography

Colour

Letters

Other

-

45%

51%

24%

4 Typography

Colour

Letters

Other

-

77%

5%

18%

5 Typography

Colour

Letters

Other

78%

26%

-

7%

Two third of the children (67%) achieved to match the image of the birthday

cake to the corresponding phrase ‘the land of birthday’. Their choices were mainly

driven by typography (“there are two candles”; “I saw the cake and I found the letter

of the cake [i]”) and by colour (“there is a yellow accent”; “there is red here and red

there”). Most children’s false matching (47%) was again based on colour associating

the image to phrases abounded with it, such as ‘the land of Christmas’ or ‘the land of

light’. A disparity of justifications was given by children for the second image which

was correctly matched by 56% of the sample, a rather low percentage. While some

children took notice of the fact that the last word (‘dance’) diverged from the usual

linear alignment of sentences in print texts reminding them of the concept of dancing

(“the letters are moving”: “[the letters] are going up and down”), other justifications

such as colour, lettering and various answers (“it’s the last one”) were also equally

mentioned. Most false answers were based on colour. Children were mainly attracted

by the colour which, in this case, led to false answers. Mostly younger children failed

to do the right matching. Relying on previous research (Papadopoulou 2001b), it is

known that children find it difficult to accept the non linear alignment of a sentence,

as most young children consider that writing has a horizontal layout). Letters and

colour were the main modes that guided children’s attention to the matching of the

third image. Twenty nine children made the right match and half of them justified

their answers based on the knowledge of some letters appearing in the phrase (“there

is an ‘X’”). An almost equal number of children referred to colour (“there is green

here and green there”; “the letters are green and red”). False matching also indicated

the ways children use to approach the visual mode. Some children mistakenly

matched the image of the Christmas tree with the phrase ‘the land of music’ referring

to the note over the letter ‘o’ associating Christmas with charades and songs. The

image of the sun was easily associated with the ‘land of light’ by most children who

used the mode of colour to justify their matching (77%). Few children made false

matching also referring to colour but relating the ‘sun’ to other colorful phrases, such

as ‘the land of birthday’. Last but not least, most children, who correctly related the

image of the piano to the phrase ‘the land of music’, noticed the transformation of the

letter ‘o’ to a music note.

Additionally, in order to investigate whether students’ correct answers

associated with their age, one-way anova analysis was performed. The total number of

children’s correct answers was used as the dependent measure. The results revealed

that, though children aged five seemed to report more elements (M = 3.5, SD = 1.5)

than younger children aged four (M = 3.05, SD = 1.5), this difference was not found to

be statistically significant, F (1, 44) = .97, p > .05.

4. Discussion

The main aim of the study was to explore whether preschoolers (four and five

year old) who had not developed formal reading skills could derive meaning from

contemporary multimodal texts based on information provided in various semiotic

modes (such as, colour, typography or other modes); another aim was to investigate

which of the semiotic modes contributed most to the meaning-making process. The

age variable was also examined in this study.

According to the findings of the present study, it seemed that preschool

children were able to detect the information provided by various modes and use it as a

cue to reveal the linguistic content of the phrases. More than two thirds of the children

were able to answer the matching activity correctly based on information provided in

various semiotic modes; in addition, they were able to provide a proper justification in

each case. In particular, the results indicated that the mode of colour, followed by

typography were mainly identified by children as the modes that contributed to the

visual salience of the phrases and helped most of them answer correctly. Regarding

the age variable, though it was found that the group of older children seemed to

perform better than the younger group, this difference was not statistically significant

requiring further research.

Overall, it was shown that the visual elements which deliberately blurred the

boundaries between image and letter were identified by most children in the attempt

to match the images with the correct phrases. In fact, the presence of colour and the

rather ‘awkward’ typographic features of the words have attracted children’s attention

in conjunction with their pre-existing familiarity with some letters of the alphabet

(usually those appearing first in a word) due to the activities conducted in the nursery

schools. The latter strategy is often used by pre-schoolers, who have not developed

formal reading skills yet, in the attempt to have an early access to literacy

(Papadopoulou & Poimenidou 2004). Few references were made to other reasons,

such as ‘the only card left’ or ‘it is the smallest or the biggest card’.

There is no doubt that any deviation from the conventional typographic form

of text makes the text impressive, funny, and familiar to pre-schoolers, as texts

intended for this age are multimodal, in which the visual mode, especially colour,

typography and images prevail. In fact, preschoolers, who have not developed full

literacy skills yet, tend to draw on the visual mode when approaching written texts;

this tendency should be cultivated and enhanced by educators, as nowadays children,

even from an early age are exposed to a variety of multimodal texts where the various

modes of communication interact to produce meaning (Baldry and Thibault 2006;

Kress and van Leeuwen 2001). Yet, in most cases, school curricula and educators

draw more on language ignoring the contribution of other modes to the meaning-

making process. In this context, there should be a shift of interest from language to

the development of a multimodal literacy. As Martens et al. (2012: 293) advocated:

“helping children read picture books multimodally and expanding their

understandings of text not only enriches their reading, but also moves them toward

being multimodal readers and creators of meaningful texts in our world”.

Taking everything into account, there seems to be an urgent need for educators

and those who plan educational policy to familiarize students with multimodal texts

enabling them to analyze, produce and critically approach information conveyed in

various modes. Students have to be taught the processes involved in constructing

meaning from multimodal texts since they are increasingly prevalent in everyday life.

In this way, educators should teach students to take advantage of the various semiotic

modes of communication in order to help them have better access to literacy and meet

the communicational demands of the era (Kress and van Leeuwen 2001).

5. References

Arizpe, Eveline and Styles Morag. 2003. Children reading pictures. Interpreting visual

texts. London and New York: Routledge Falmer.

Baldry, Anthony and Thibault Paul John. 2006. Multimodal transcription and text

analysis. London & Oakville: Equinox.

Charmaz, Kathy. 2000. Grounded theory: Objectivist and constructivist methods. In

Norman Denzin & Yvonna Lincoln (eds.), Handbook of qualitative research,

769-802. California: SAGE Publications.

Cope, Bill and Kalantzis Mary. 2000. Introduction: Multiliteracies: The beginnings of

an idea. In Bill Cope & Mary Kalantzis (eds.), Multiliteracies: Literacy

learning and the design of social futures, 3-8. New York: Routledge.

Fleckenstein, Kristie. 2002. Inviting imagery into our classrooms. In Kristie

Fleckenstein, Lina Calendrillo & Demetrice Worley (eds.), Language and

images in the reading-writing classroom, 3-26. Mahwah, NJ: Erlbaum.

Kalantzis, Mary and Cope Bill. 2012. Literacies. Australia: Cambridge University

Press.

Kiefer, Barbara. 1995. The potential of picturebooks: From visual literacy to aesthetic

understanding. Englewood Cliffs, NJ: Prentice Hall.

Kress, Gunther, Jewitt Carey, Ogborn Jon., and Tsatsarelis Charalambos. 2001.

Multimodal teaching and learning: The rhetorics of the science classroom.

London: Continuum.

Kress, Gunther and Van Leeuwen Theo. 2001. Multimodal discourse: The modes and

media of contemporary communication. London: Arnold.

Kress, Gunther and van Leeuwen Theo. 2006. Reading images: The grammar of

visual design. London: Routledge

Marantidou, Xara. 2010. Εγώ + Εσύ = Μαζί. Η Τέχνη του να Είσαι Γιαγιά [Me + you

= together. The art of being a grandmother]. Athens: Papyros.

Martens, Prisca, Ray Martens, Doyle Michelle Hassay, Jenna Loomis & Stacy

Aghalarov. 2012. Learning from picture books: Reading and writing

multimodally in first grade. The Reading Teacher 66(4). 285–294.

Meek, Margaret. 1988. How texts teach what readers learn. Stroud, England: Thimble

Press.

New London Group. 1996. A pedagogy of multiliteracies: Designing social futures.

Harvard Educational Review 66(1). 60-92.

Nunan, David. 1992. Research methods in language learning. United States of

America: Cambridge University Press.

Pantaleo, Sylivia. 2005. Reading young children’s visual texts. Early childhood

research and practice, 7(1). http://ecrp.uiuc.edu/v7n1/pantaleo.html (accessed

10 January 2011).

Papadopoulou, Maria. 2001a. Multimodality as an access to writing for preschool

children. International Journal of Learning 8. 1-15.

Papadopoulou, Maria. 2001b. Αντιλήψεις και Αναπαραστάσεις των Παιδιών

Προσχολικής Ηλικίας για τη Γραφή [Ideas and representations of young

children about writing]. In P. Papoulia-Tzelepi (ed.), Emergent literacy.

Research and practices, 97-120. Athens: Kastaniotis.

Papadopoulou, Maria & Maria Poimenidou. 2004. ∆ιαβάζουµε τα γράµµατα ή τα

µετράµε; Οι στρατηγικές των παιδιών προσχολικής ηλικίας για να

προσεγγίσουν ένα γραπτό κείµενο [Reading or counting letters: Preschool

children's strategies while approaching a text]. In N. Zoukis & Th. Mitakos

(eds). Proceedings of the 3rd Congress of the Pedagogical Society of Greece.

Retrieved from

http://www.pee.gr/wpcontent/uploads/praktika_synedrion_files/e_athena/sin_a

th.htm

Papadopoulou, Maria, Manoli Polyxeni & Zifkou Elisavet. 2014. Typography: How

noticeable is it? Preschoolers detecting typographic elements in illustrated

books. The International Journal of Early Childhood Learning 20(2). 23-36.

Patton, Michael Quinn. 1990. Qualitative evaluation and research methods. Newbury

Park: SAGE Publications.

Serafini, Frank & Jennifer Clausen. 2012. Typography as semiotic resource. Journal of

Visual Literacy 31(2).1-16.

Styles, Morag & Arizpe Evelyn. (2001). A gorilla with “ grandpa's eyes ”: How

children interpret visual texts-a case study of Anthony Browne's Zoo. Children's

Literature in Education 32(4). 261-281

Unsworth, Len. 2001. Teaching multiliteracies across the curriculum: Changing

contexts of text and image in classroom practice. Great Britain: Open

University Press.

Walker, Sue & Linda Reynolds. 2003. Serifs, sans serifs and infant characters in

children’s reading books. Information Design Journal 11(3). 106-122.

Walsh, Maureen. 2003. Reading pictures: What do they reveal? Young children’s

reading of visual texts. Reading 37(3). 123-130.

Yannicopoulou, Angela. 2004. Visual aspects of written texts: Preschoolers view

comics. L1- Educational Studies in Language and Literature 4. 169-181.