Full-length paper: Image selection and annotation for an environmental knowledge base

Arianne Reimerink (corresponding author)University of GranadaC/ Buensuceso 1118002 GranadaSpainEmail: arianne@ugr.esTel: +34 685 107291Fax: +34 958 244104

Pilar León-AraúzUniversity of GranadaC/ Buensuceso 1118002 GranadaSpain

Pamela FaberUniversity of GranadaC/ Buensuceso 1118002 GranadaSpain

AbstractImages play an important role in the representation and acquisition of specialized knowledge. Not surprisingly, terminological knowledge bases (TKBs) often include images as a way to enhance the information in concept entries. However, the selection of these images should not be random, but rather based on specific guidelines that take into account the type and nature of the concept being described. This paper presents a proposal on how to combine the features of images with the conceptual propositions in EcoLexicon, a multilingual TKB on the environment. This proposal is based on the following: (i) the combinatory possibilities of concept types; (ii) image types, such as photographs, drawings and flow charts; (iii) morphological features or visual knowledge patterns (VKPs), such as labels, colours, arrows, and their effect on the functional nature of each image type. Currently, images are stored in association with concept entries according to the semantic content of their definitions, but they are not described or annotated according to the parameters that guided their selection, which would undoubtedly contribute to the systematization and automatization of the process. First, the images included in EcoLexicon were analyzed in terms of their adequateness, the semantic relations expressed, the concept types and their VKPs. Then, with these data, guidelines for image selection and annotation were created. The final aim is twofold: (1) to systematize the selection of images and (2) to start annotating old and new images so that the system can automatically allocate them in different concept entries based on shared conceptual propositions.

KeywordsKnowledge representation, Image selection, Image annotation, EcoLexicon

1

1. Introduction

EcoLexicon (ecolexicon.ugr.es) is a multimodal, multilingual terminological knowledge base (TKB) on the environment in which concepts are interrelated, based on the information extracted from a specialized domain corpus created for EcoLexicon. However, the way concepts are related can also be reflected in the graphical images depicting these concepts. For this reason, a visual corpus was also compiled to enrich conceptual description in EcoLexicon. Currently, images are stored in association with concept entries according to the semantic content described in their definition, but they are not described or annotated according to the parameters that guided their selection, which would undoubtedly contribute to the systematization and automatization of the process.

Images, as a type of communicative sign, need to be analyzed in greater depth. In the same way as language, they have a combinatorial potential as well as a grammar (Faber 2012; Prieto Velasco 2008; Prieto Velasco and Faber 2012). An image grammar is based on the relationship between visual and verbal communication since both types of communication overlap in many intermedial contexts (Nöth 2001: 2). Accordingly, if visual images are signs, theories of depiction should contain semiotic theory elements with a syntactic, semantic, and pragmatic component (Scholz 2000: 202).

Image annotation is often defined as the labelling of the semantic content of images with a set of keywords (Wenyin et al. 2001). However, “even though an image is worth a thousand of words, humans still possess the ability to summarize an image’s contents using only one or two sentences. Similarly, humans may deem two images as semantically similar, even though the arrangement or even the presence of objects may vary dramatically” (Zitnick and Parikh 2013).

Manual image annotation, apart from inconsistent, can be very time-consuming. For this reason, in computer science automatic image annotation has been studied for some time now (Jeon et al. 2003; Li and Wang 2008). However, these studies mostly focus on photographs (Zitnick and Parikh 2013 being one of the exceptions), objects and rather general concepts. Furthermore, they do not take into account the interaction of the semantic elements. In this sense, Mei et al. (2008) acknowledge that approaches to automatic image annotation do not usually guarantee good semantic coherence of the annotated words for each image, because they treat each word independently without considering the inherent semantic coherence among the words. Unfortunately, given the specificity of the graphical information that an environmental TKB requires, such procedures cannot be applied in our case. Nonetheless, the dataset resulting from image annotation in EcoLexicon could be used in the future to train automatic image annotation systems based on the features that may be of interest in the graphical representation of specialized conceptual knowledge.

Image annotation forces us to clearly think about naming and categorization issues (Barriuso and Torralba 2012), which are tasks that are not as straight-forward as they may seem. For this reason, guidelines need to be provided for the future annotators. This paper proposes a method of combining the features of images with the conceptual propositions in EcoLexicon, based on the following: (i) concept types; (ii) image types, such as photographs, drawings and flow charts; (iii) morphological features or visual

2

knowledge patterns (VKPs), such as labels, colours, arrows, and their effect on the functional nature of each image type.

The combinatorial potential of concept types is discussed in Section 2 of this paper. Section 3 explains how perspective and context affect this potential and its visual representation. Section 4 analyzes the morphological and functional criteria of images for knowledge representation. Section 5 describes the evaluation process of EcoLexicon images and, following the results of this study, Section 6 proposes image selection and annotation guidelines with the minimum descriptive requirements for these images in our database. The final aim of these guidelines is twofold: (1) to systematize the selection of images and (2) to start annotating old and new images so that the system can automatically allocate them in different concept entries based on shared conceptual propositions.

2. Conceptual knowledge in EcoLexicon

The knowledge extracted from the EcoLexicon corpus is organized in a frame-like structure or prototypical domain event, namely, the Environmental Event (EE) (Figure 1, Faber et al. 2007; León Araúz et al. 2009; Reimerink and Faber 2009). This prototypical domain event or action-environment interface (Barsalou 2003) provides a template applicable to all levels of information structured from a process-oriented perspective (Faber et al. 2006; Reimerink and Faber 2009; León Araúz et al. 2009).

The EE is conceptualized as a dynamic process. This process is initiated by an agent (either natural or human), affects a specific kind of patient (an environmental entity), and produces a result in a certain geographical area (see Figure 1). These macro-categories (AGENT, PROCESS, and PATIENT/RESULT) are the semantic roles inherent to this specialized domain. Accordingly, the EE is a model that represents interrelationships at different levels.

Fig. 1 Environmental Event (EE)

3

In EcoLexicon, concepts appear in networks that link them to other concepts by means of a closed inventory of semantic relations. This inventory is especially conceived for Environmental Science and includes its most typical semantic roles. Figure 2 shows the network for DAM. This three-level network represents the many ways that DAM is linked to other concepts by means of vertical relations (type_of, part_of, etc.) and horizontal relations (has_function, affects, etc.).

Fig. 2 Conceptual network of DAM

Based on the data in the EcoLexicon corpus, conceptual relations depend on concept types and nature. Table 1 shows the relation types associated with the elements that can be linked in conceptual propositions (León Araúz 2009; León Araúz and Faber 2010). Each of these relations also has a corresponding inverse relation (part_of: has_part; causes: has_agent, etc.) except for delimited_by, which is the only bi-directional relation in the inventory.

Conceptual relations Concept 1 Concept 2 Conceptual

propositionType_of Physical entity

Mental entityProcess

Physical entityMental entityProcess

Masonry dam type_of dam

Part_of Physical entity

Mental entity

Physical entity

Mental entity

Main layer part_of breakwater

microbiology part_of biology

4

Phase_of Process Process pumping phase_of dredging

Made_of Physical entity Physical entity air made_of gasHas_location Physical entity Physical entity jetty located_at canalTakes_place_at Process Physical entity littoral transport

takes_place_at seaDelimited_by Physical entity Physical entity stratosphere

delimited_by stratopause

Result_of Process Process aggradation result_of sedimentation

Causes Physical entity Process water causes erosionAffects Physical entity

Mental entityPhysical entityMental entity

Process

Process

Process

Entity

Entity

Process

groyne affects littoral transportpesticide affects water

wave affects groyne

precipitation affects erosion

Has_function Entity Process aquifer has_function human supply

Attribute_of PropertyProperty

EntityProcess

abyssal attribute_of plainanthropic attribute_of process

Table 1. Relation types

Meronymy is divided into six subrelations (part_of, phase_of, made_of, has_location, takes_place_at, delimited_by) since not all parts interact in the same way with their wholes. This distinction is in consonance with the subtypes proposed in Winston et al. (1997) and is based on domain-specific needs, mainly on ontology reasoning and consistency. For example, if has_location were considered to be a part_of relation, this would cause fallacious transitivity (Murphy 2003). If a SPILLWAY were part_of a DAM and a DAM part_of a RIVER, an ontology would infer that SPILLWAYS are part_of RIVERS, which is highly implausible. In the same way, if both processes and entities were connected by the same part_of relation, there would be no restrictions on category membership or disjunction (León Araúz and Faber 2010). This distinction is also useful for image categorization (see Section 4) since images can depict meronymy in very different ways.

This combinatorial potential is an indication of certain constraints associated with the nature of the concept. However, these combinations are also constrained by context (Section 3). The most recurrent concepts in the domain (physical entities and processes) are the ones that can be linked to others by means of a wider variety of relations. Nevertheless, there are also certain relations that are exclusive of a single type, such as attribute_of (for properties), studies (for sciences and disciplines), and measures (for instruments). These relations are all subtypes of has_function. Although concept nature triggers or restricts the activation of a set of possible relations, at the same time it also determines which other concept types can be linked by each relation. For instance, a

5

process may activate affected_by, but only if it is associated with a physical entity. However, if it activates its inverse relation, affects, it can be linked to entities, events, and properties. These conceptual constraints are also reflected in images (see Section 4).

3. Semantic networks and contextual domains

Multidimensionality (Roger 2004; Kageura 1997) is commonly regarded as a way of enriching traditional static representations and of enhancing knowledge acquisition by providing different points of view in the same conceptual network (León Araúz and Faber 2010). Nevertheless, multidimensionality in the environmental domain can be an important source of information overload. This is especially true of versatile concepts, such as WATER (see Figure 3), which are usually top-level concepts that participate in a myriad of events. For instance, WATER is linked to natural and artificial processes, such as EROSION or DESALINATION; and its subtypes range from cloud-related water (e.g. SUPERCOOLED WATER) to human-derived water (e.g. WASTEWATER).

Fig. 3 Information overload in the network of WATER

However, WATER rarely, if ever, activates all these relations at the same time since they evoke completely different situations. Our assertion is that any specialized domain contains sub-domains in which conceptual dimensions become more or less salient, depending on the activation of a specific frame. As a result, a more believable representation system should account for recontextualization as reflected in the situated nature of concepts. In EcoLexicon, this is accomplished by dividing the environmental field into discipline-based contextual domains (see Figure 4).

6

Fig. 4 Ecolexicon contextual domains

Contextual constraints are not applicable to individual concepts or to individual relations. Rather they are applied to the conceptual propositions derived from corpus analysis. For example, processes, such as PUMPING or AERATION, are linked to WATER by the affect relation. Nonetheless, these propositions are not relevant if users are only interested in the way that WATER interacts with the landscape. Consequently, such propositions only appear in the context of Water Treatment and Supply. When constraints are applied, the network of WATER within this domain is recontextualized and becomes more meaningful (see Figure 5). This restricts the combinatorial potential of WATER with other concepts. The prominence of the attribute_of and affects relations highlights the concept as a patient of artificial processes.

7

Fig. 5 WATER in the Water Treatment and Supply domain

In the same way that context constrains the conceptual relations activated in recontextualized networks, images can also focus on certain features of the concepts and relations that they depict.

The images in Figure 6 show different facets of WATER. The top left-hand corner (6.1) of the image focuses on the function of WATER since it depicts WATER as a drinkable liquid and the instrument (GLASS) used for this purpose. The bottom left-hand corner (6.4) of the image indicates the part_of relation since the molecule shows the composition of WATER. This also has a direct implication in the dynamic categorization of the concept, since image (6.1) categorizes WATER as a resource and image (6.4) categorizes it as a chemical compound.

Evidently, the frame and background in which WATER is portrayed also play a role. The outfall in (6.5) and the landscape in (6.6) show WATER in different scenarios and activate different conceptual propositions in each case. For instance, in (6.5), the outfall activates the has_function relation in the conceptual proposition OUTFALL transports WASTEWATER.

In contrast, images (6.2) and (6.3) do not point to other concepts. However, since they represent WATER as a DROPLET and as a spiral during the CORIOLIS EFFECT, it would be more suitable to include them in the entries of these concepts.

8

Fig. 6 From left to right and top to bottom, images (6.1), (6.2), (6.3), (6.4), (6.5), and (6.6) of WATER

Since all these images closely resemble their real-world object (except for 6.4), they show a high level of referential similarity. However, there are a wide variety of image types (e.g. photographs, flowcharts, drawings, etc.) with an array of other visual knowledge patterns (VKPs) such as colours, textual elements, arrows, etc. Since so far the selection of images has been a rather intuitive process, our current aim is to classify such features in terms of the natural and contextual combinatorial potential of the concepts. We will first analyse how this has been done in EcoLexicon both in qualitative (Section 4) and quantitative terms (Section 5). Based on these data, we will derive a set of unified criteria for the future selection and annotation of the images (Section 6).

4. Selection criteria for images

Traditionally, images have been classified according to their morphology. The usual categories are photographs, drawings, animations, videos, diagrams, charts, graphs, schemas, views, etc. (Darian 2001; Monterde 2002). However, images can also be classified in terms of their most salient functions (Anglin et al. 2004) or in terms of their relationship with the real-world entity designated. Prieto Velasco (2008), Faber et al. (2007), and Prieto Velasco and Faber (2012) applied the functional criteria of iconicity, abstraction, and dynamism to the representation of specialized concepts.

More specifically, iconic images resemble the real-world object represented through the abstraction of conceptual attributes in the illustration. In contrast, abstraction in an illustration is a matter of degree, and refers to the cognitive effort required for the recognition and representation of the concept (Levie and Lentz 1982; Park and Hopkins 1993; Rieber 1994). Finally, dynamism is the explicit or implicit representation of movement in time and space. For example, videos and animations are explicitly dynamic, whereas static images portraying different stages of a process are implicitly dynamic (Prieto Velasco 2008; Reimerink et al. 2010).

These functional criteria add a conceptual dimension to the classification of images rather than solely focusing on their form and have been applied to the selection of images in EcoLexicon until recently. An example of how this is done is shown in Table

9

2. In EcoLexicon, definitions are based on templates that define category membership. These definitional templates are combined with images that further explain the relations in the templates. The linguistic description of the concepts in EcoLexicon follows these templates insofar as the type, quantity, and configuration of information is concerned. In this way, definitions have a uniform structure that complements the information in conceptual networks, and directly refer to and evoke the underlying conceptual structure of the domain. These templates can be considered a conceptual grammar that thus ensures a high degree of systematisation (Montero and García 2004; Faber et al. 2007; Faber 2012).

Water erosion[Type_of] Erosion

[Has_agent]Water(river, stream, rain, wave, current…)

[Affects]Earth’s surface(beaches, mountains, soil…)

[Has_result]

SheetRill

Gully

Cliff

Beach

[Has_phase]

Weathering

Transport

10

Deposition

Table 2. WATER EROSION template (León et al. 2013)

In the case of WATER EROSION, the template includes the four basic relations of all natural processes: is_a, has_agent, affects and has_result. It also has an additional relation because it is a subtype of a complex procedural concept, EROSION, which can be divided into a sequence of steps: has_phase.

The template of WATER EROSION constrains the has_agent relation to WATER, whereas its hypernym EROSION has many possible agents: WATER, WIND, ICE, GRAVITY, and ANIMALS. Given that the has_patient relation is the Earth’s surface and all its subordinates, it is difficult to show in only one image. The has_phase relation is the same as the superordinate EROSION. Here, three images have been added showing the three phases of WATER EROSION. Rather than the phases themselves, these images photographically portray the result of each of these phases. The concept is a natural process and in this case, the representation of complex events conveyed by relations, such as has_phase, can be conveyed in photographs. What is important in this entry is the final result of the processes inherent in WATER EROSION rather than how the phases of each subprocess (WEATHERING, TRANSPORT and DEPOSITION) occur. Nevertheless, the only way to show the propositions, WATER EROSION has_result SHEET/RILL/GULLY, and the fact that these three are spatiotemporally linked to each other is by a more abstract image such as the one in Table 2. This image condenses the information and at the same time shows the relatedness of these land formations.

The definitional template provides a systematic means to select images for each concept entry in the TKB. Images are thus regarded as a whole and are only linked to the concept itself. So far, we have shown how the same concept can be (and most often should be) represented through different images, depending on perspective, or the semantic content highlighted. However, the same image can also work for the representation of other related concept entries (e.g. an entity and the process through which it was formed, a concept and its parts, etc.). Images should thus be further dissected according to the different types of knowledge they convey (i.e. image type, morphological features, semantic content, etc.). They usually show several concepts in a specific background where they establish different relations that can be explicitly labelled or inferred from previous knowledge. In this sense, we believe that images should not be stored in the TKB as the representation of a concept, but as the representation of a set of conceptual propositions and that they should be annotated according to semantic and morphological information. Since each image activates several propositions and each proposition can be activated by different concepts, one image would then appear in several concept entries. This would enhance the reusability of images, improving the consistency of the TKB and avoiding duplicating workload. Moreover, this would provide the means to recontextualize images much like the recontextualization of semantic networks in Section 3. In this way, images will be

11

shown or not to the end-user depending on the activated context and the same image can illustrate all contexts where the same proposition comes up.

The qualitative part of the present study combined both morphological criteria and the functional criteria referential similarity and dynamism to analyse visual knowledge patterns (VKPs) in images. Referential similarity refers to the degree to which an image resembles its referent in the real world. This similarity is measured on a continuum ranging from non-similar to totally identical. Consequently, referential similarity can be regarded as the convergence of iconicity and abstraction. It goes without saying that a two-dimensional image can never be totally identical to its referent, but a colour photograph would have a high degree of referential similarity. Dynamism can also be measured on a continuum ranging from totally static to very dynamic. The analysis showed which VKPs and which degrees of referential similarity and dynamism are most characteristic of different types of images and how they are related to the conceptual propositions represented in each one.

Depending on whether a concept is an entity or a process, images have different characteristics. More specifically, the VKPs in an image as well as its referential similarity and dynamism vary, depending on the conceptual propositions activated.

4.1 Visual representation of entities

Figure 7 is a colour photograph of a DAM, which has the maximum level of referential similarity. The image is static and it portrays the basic properties that define DAM, namely, its location and the difference in water level on both sides of the dam, etc. The image thus provides an overall view of the nature of the object and is the best way of illustrating the type_of relation. In this case, the photograph accurately portrays DAM as a water-retaining structure.

12

Fig. 7 Static image of DAM with high level of referential similarity: type_of relationSource : http://pinker.wjh.harvard.edu/photos/american_west/images/Glen%20Canyon%20Dam%20from%20Bridge.jpg

However, this image would not be the best way of showing the parts of a dam. For this purpose, Figure 8 is more appropriate. Although the schematic image is more abstract with a lower degree of referential similarity, the VKPs, labels and arrows pointing to the different parts of the dam, make the part_of relation explicit to lay users querying EcoLexicon.

Fig. 8 Static image for conceptual propositions parapet walls, free board, gallery, heel, toe, sluice way, crest and spillway part_of damSource: http://civilsolution.wordpress.com/tag/spillway/

Figure 9 shows how the made_of relation can be conveyed. Even though an object is composed of a certain material, this relation differs from the meronymy in Figure 8 because the same structure, for example, can be constructed from different types of material. For example, a GROYNE HEAD is part_of all groynes. However, a GROYNE can be made_of STONE, CONCRETE, or WOOD. Accordingly, the illustration of this relation requires more than one image and requires images with a higher level of referential similarity.

13

Fig. 9 Static image for GROYNE made_of WOODSource: http://blog.seamaidengemsjewellery.co.uk/

Figure 9 can also be used to convey GROYNE has_location COAST. This relation is relevant when the location of a physical object is essential for its description. For instance, a GROYNE is not a GROYNE if it is not located on the COAST.

In addition, more schematic images can also be used to show other location-related attributes, such as position or orientation (see Figure 10). Figure 10 also illustrates the type_of relation of GROYNE, because it highlights shape, which is the property that differentiates different subtypes of GROYNE (STRAIGHT GROYNE, T-SHAPED GROYNE, L-SHAPED GROYNE, etc.).

Fig. 10 groyne has_location coast; groyne has_attribute perpendicularSource: http://geoaphyprojectrules.blogspot.com.es/2012/07/groynes.html

The images selected for inclusion in a concept entry should be the ones that best represent the basic properties and relations that define the concept in relation to others. Depending on the nature of the relation and the concepts, certain types of image

14

accomplish this better than others. This is the case of Figure 8, which portrays the part_of relation for DAM more explicitly than Figure 7.

Another example is BEACH. Beaches are made of pebbles and sand, which are usually mixed together with no clear-cut boundaries to separate them. Despite the fact that Figure 9 shows a beach with pebbles and sand, it would be a poor choice since the focus is on the defence structure and the beach, rather than the material that the beach is made of. An image focusing on the made_of relation should have no object in the picture to distract the viewer’s attention, as occurs in Figure 11.

Fig. 11 beach made_of sand/pebblesSource: http://www.dreamstime.com/stock-image-pebble-beach-waves-image2462571

As previously mentioned, static images with a high level of referential similarity are often sufficient to represent a location. Nevertheless, if the relation focuses on artificial divisions (e.g. the layers of the atmosphere), additional textual information is required. In this case, the conceptual relation is delimited_by. This is a domain-specific relation, mainly for geographic entities, such as STRATOSPHERE and MESOSPHERE, which are delimited_by STRATOPAUSE (Figure 12).

Fig. 12 Static image for the relation delimited_bySource: http://www.geologywales.co.uk/storms/summer09c.htm

Figure 12 can be reused for other conceptual relations as well. More specifically, it shows the properties of the different atmospheric layers and also represents the type_of relation. Concepts, such as atmospheric layers, whose comprehension depends on their comparison or contrast, are best represented in schematic images in which the VKPs increase the image’s explanatory power.

15

The VKPs in Figure 12 are the following: (i) contrasting colours bordered with black lines to highlight the boundaries between the atmospheric layers; (ii) labels for each layer; (iii) short explanations of the properties of each layer; (iv) the separation of the layers, and their heights; (v) a tiny picture of a thunder storm along with its label to indicate that all weather-related phenomena take_place_in the TROPOSPHERE.

4.2 Visual representation of processes

Processes are generally described by the meronymic relations takes_place_in and phase_of because processes are composed of different stages and occur within a certain context. This is in direct contrast to physical objects, whose description is dominated by the relations has_location and part_of. Not surprisingly, processes are generally portrayed by flow charts that represent more than one relation. For example, Figure 13 is an image of the GEOLOGICAL CYCLE, an extremely complex process, which shows both the take_place_in and phase_of relations.

Fig. 13 Geological cycleSource: http://finstone.fi/engl/geology/

As can be observed in Figure 13, the concepts HARDENING, METAMORPHISM, MELTING, CRYSTALLIZATION, and INTRUSION take_place_in under the EARTH’S SURFACE. At the same time, they are also phases_of GEOLOGICAL CYCLE. Figure 13 also conveys the

16

result_of relation. This relation is relevant to either events or entities that are derived from other events. In this case, it shows SEDIMENTARY ROCKS result_of HARDENING, METAMORPHIC ROCKS result_of METAMORPHISM, etc. A comparison of the WATER CYCLE (Figure 13) and GEOLOGICAL CYCLE (Figure 14) shows that these representations of certain natural objects and events (e.g. SUN, RAIN, CLOUDS, MAGMA, VOLCANIC ERUPTION, etc.) have a high degree of similarity to their referents in the real world.

Fig. 14 The WATER CYCLESource: http://sageography.myschoolstuff.co.za/geogwiki/grade-10/the-atmosphere/moisture-in-the-atmosphere/

However, in Figure 13, other less well-known objects are labelled to explain where one type of geological formation ends and the other begins. This is the case of SEDIMENTARY ROCKS, METAMORPHIC ROCKS, MAGMA, and IGNEOUS ROCKS. Although referential similarity characterizes the background in Figure 14, the objects and processes of the WATER CYCLE are explicitly labelled (SURFACE RUNOFF, WATER TABLE, etc.). In both images the names of complex processes are also included (e.g. METAMORPHISM, CRYSTALLIZATION, EVAPORATION, CONDENSATION, etc.).

Furthermore, in both images the use of similar yet different colours heightens resemblance and, at the same time, delimits similar concepts or those that occur in connected locations. More specifically, water and sky are different shades of blue, and there is a gradual colour change from yellow to red and dark brown to show how sediment becomes rock and then magma (Figure 13). Different shades of green, yellow, and brown indicate where processes, such as INFILTRATION and PERCOLATION, occur (Figure 14). Arrows add dynamism to the images that portray how certain processes stem from others and how they affect one another. Thus, arrows as visual knowledge patterns (VKPs) most often convey meronymy in the case of entities (part_of) and (phase_of), and the result_of relation in the case of processes.

17

The blue arrows rising from the sea in Figure 13 represent EVAPORATION. Since this process is part of the WATER CYCLE, it is not labelled. Consequently, Figure 13 is a more suitable image for the description of GEOLOGICAL CYCLE and Figure 14 is more appropriate for the WATER CYCLE.

Other complex concepts include natural objects that in certain contexts can be conceived as having a function. Although such entities are not necessarily artefacts, they can be used for human profit. Their representation must thus highlight the has_function relation. Natural concepts with a function are, for instance, AQUIFER (has_function WATER SUPPLY) or SAND (has_function BEACH NOURISHMENT). In Figure 15, the function of aquifers (i.e. urban and agricultural recharge) is labelled accordingly as well as transmitted by other VKPs. Labels show the dual function of aquifers (both for urban and agricultural uses), whereas VKPs such as arrows and a certain level of referential similarity show the location where the event takes place, the instruments with which aquifers are exploited, and the cyclical nature of the process: from ground to surface and back to the ground.

Fig. 15 AQUIFER has_function WATER SUPPLYSource: http://pubs.usgs.gov/fs/2005/3022/

4.3 Visual knowledge patterns

In computational approaches to image annotation and retrieval, visual features may be low level (e.g. pixel values, colour, texture and shape of segments of the image) or high level (i.e. semantics). Between them there is what is known as the semantic gap (Smeulders et al. 2000). Semantic meaning relies on the understanding of the attributes of the visible objects and their relations (Zitnick and Parikh 2013). Thus, high-level features are used in attribute-based approaches (e.g. properties of materials, spatial layouts, faces; Parikh 2013) and are usually domain-specific (e.g. cloth retailing, in Di et al. 2013). In the case of EcoLexicon, low level features correspond to image types and VKPs, whereas high level features are concepts and relations.

The analysis of the images in Sections 4.1 and 4.2 shows that VKPs are polysemic since the same pattern can be used for different purposes in the same way that textual knowledge patterns can also convey different conceptual relations (León-Araúz and Reimerink 2009). Accordingly, the conceptual knowledge underlying VKPs can only be

18

interpreted in the context of each image. Nevertheless, a certain combination of patterns constrained by image and concept types makes images more or less suitable for the representation of certain types of conceptual knowledge.

Even though an arrow can be used to connect a concept name to its representation in the image, this VKP does not necessarily transmit dynamism. In fact, arrows indicating parts or names usually do not as in Figure 8, where the arrows only point to the components of the dam. However, when arrows appear in an image representing a process, they generally convey dynamism (see Figures 13 and 14) and go in the direction of the different phases of the process.

The same is true for colours. In images with a high level of referential similarity, the colours in the image are the same or similar to those of the real world entity. This includes colours in colour photographs as well as colours used to imitate reality in drawings and flow charts. These colours may be used to enhance the referential background of images, another VKP that shows the setting in which the entities and processes appear. In contrast, the function of the colours in Figure 12 is not to realistically represent the layers of the atmosphere, but rather to differentiate them. In Figure 13 (GEOLOGICAL CYCLE) and Figure 14 (WATER CYCLE), colour-coding is used for both purposes.

In this section several image types have been analyzed: photographs, drawings and flow charts. Photographs (Figures 7, 9, and 11) are characterized by their static nature and a high level of referential similarity and thus referential background. They are useful for representing the type_of relation for physical entities; the made_of relation, where a physical entity and the material that it is made of are closely related; and the has_location relation, where the physical entity is shown in its physical environment.

Drawings (Figures 8, 10 and 12) are defined by their lower level of referential similarity and their combination with VKPs such as labels and arrows that point to different parts or sections of the conceptual proposition represented. They are most frequently used to represent meronymy in conceptual propositions such as PARAPET WALLS part_of DAM and the more complex GROYNE has_location COAST, where the orientation of the groyne is represented. Drawings are also the best option to differentiate between coordinate concepts of physical entities that have different shapes (T-SHAPED GROYNE type_of GROYNE) or between other closely related concepts that are hard to differentiate without reference to the other concepts, as in the propositions in Figure 12, STRATOSPHERE and MESOSPHERE delimited_by STRATOPAUSE.

Processes are more easily represented by flow charts that combine certain VKPs. In the first place, the background should show a relatively high level of referential similarity and represent the larger context in which the processes take place. Arrows add dynamism and show the direction of the movement, or the sequence of process phases. The more complex the process or event is the more labels are added to show their relation to each other. In these flow charts, colour-coding is used both for representing a high level of referential similarity and for contrasting closely related concepts. Figures 13 and 14 are examples of how complex events can be represented in flow charts.

5. Evaluation of EcoLexicon images

19

The selection criteria described in Section 4 are the result of a qualitative analysis based on our experience working on the TKB EcoLexicon. These are certain generalizations all our researchers can agree on when selecting images to feed the TKB, however, a quantitative analysis is also necessary in order to confirm our experience and to create guidelines for both image selection and annotation based on semantic content (Section 6).

EcoLexicon images were analyzed in terms of (1) their adequateness in representing one or more conceptual propositions linked to the concept entry, (2) the reusability of the image, (3) the semantic relations expressed, (4) the concept types involved, and (5) the VKPs used to convey the information. The images and the concepts they are linked to were exported to a spread sheet, where they were manually assigned a number according to the level of adequateness: 0, not at all; 1, partially; and 2, completely. Then another number was assigned according to the reusability of the image (0, no; 1, yes). Other columns included information on the image type (photograph, drawing – including maps and diagrams – or flow chart), semantic content (concepts and relations) and VKPs (labels, arrows, colours and their specific use). The assessment outcome is explained with the following example:

Fig. 16 Drawing in concept entry BARRIER ISLANDSource: http://www.beg.utexas.edu/UTopia/contentpg_images/gloss_barrier_island2.jpg

Concept Image type Conceptual propositions

VKPs Adequate Reusability

Barrier island

Drawing Barrier island has_location nearshoreBarrier island has_part back barrier/ foredune/ beach

Static imageColours (referential similarity and distinction)Arrows (denomination

2 1 (foredune, tidal flat, surfzone, etc.)

20

Barrier island delimited_by tidal flat/nearshoreetc.

of parts)

Table 3. Assessment outcome for drawing in concept entry BARRIER ISLAND

In this example, Figure 16 is found in the entry for BARRIER ISLAND and it represents several conceptual propositions. It has therefore been assigned a 2 (see Table 3), for it is fully adequate. The combination of image type (drawing) and VKPs (colours that provide a high level of referential similarity and arrows with labels) makes it especially adequate for the representation of the part_of relation (see Section 4). In this specific case, the image represents more conceptual propositions (has_location and delimited_by) because of the larger context in which the concept is shown. It can also be easily reused in entries of the parts of the concept and, again because of the larger context, in geographically related concepts such as TIDAL FLAT and SURF ZONE.

5.1 Adequateness and reusability

Currently, EcoLexicon includes 3599 concepts and 1113 concepts have one or more images linked to their entry (31%). The total number of images amounts to 1698. Of these, 90.8% are adequate, 8.0% are partially adequate and 1.2% are not. Some are only partially adequate because the images are too small or unclear in the sense that you cannot see them properly, which is probably due to changes in format when introduced in EcoLexicon from other sources.

Fig. 17 Inadequate image in the concept entry for POLDERSource: http://www.dasgrafikbuero.de/nord/seiten/grohde%20polder_005.htm

21

Figure 17 is an example of an inadequate image for the concept POLDER. In this case, it seems that the person who selected the image misunderstood the concept, as what appear in the image are dunes and not a polder. These images must be replaced by more appropriate images.

Although most images are adequate in the sense that they represent at least one conceptual proposition, sometimes several very similar images that do not add distinguishable conceptual knowledge are selected for the same entry in EcoLexicon. AQUIFER is a good example with four drawings with exactly the same information. Of these, the best one should be selected and linked to all the related concepts. The rest should be discarded. On the other hand, in the concepts CONFINED and UNCONFINED AQUIFER one image (Figure 18) that perfectly distinguishes these closely related concepts has been used.

Fig. 18 Adequate image for CONFINED and UNCONFINED AQUIFERSource: http://water.usgs.gov/gotita/earthgwaquifer.html

Most images are colour photographs representing type_of and has_location relations. These images cannot be reused often, because they represent the real world object. Some of them could be reused if we take into account that these real world objects are the result of certain processes, for example the concept BAR in Figure 19 is a colour photograph that represents the conceptual propositions BAR has_location RIVER MOUTH and BAR result_of LITTORAL SEDIMENTATION.

22

Fig. 19 Adequate image for BAR has_location RIVER MOUTH and BAR result_of LITTORAL SEDIMENTATIONSource: http://www.mappinginteractivo.com/plantilla-ante.asp?id_articulo=464

Through the annotation of all conceptual propositions in these images, they can be linked to the concept entries of the related processes – and not only to those of the entities resulting from them –, which would be a way to enhance reusability.

Instruments are often depicted by colour photographs as well. This is a problem because the most important conceptual relation for instruments is has_function, which cannot be easily represented by a photograph. These images should therefore be combined with images that represent the process in which they participate, so the has_function relation is made explicit. A good example is DREDGING in Figure 20, where the takes_place_in and has_instrument relations are quite clear, but the has_function relation is not.

Fig. 20 Adequate image for DREDGING takes_place_in SEA and has_instrument DREDGESource: http://www.dragadoshidraulicos.com/

Another possibility to make the has_function relation explicit is to combine instruments with the output they provide, for example the concept entry METEOROGRAPH includes a

23

photograph of the instrument and the concept METEOROGRAM includes a photograph of the output of the instrument. If we explicitly link both images to both concept entries, the has_function relation will be much clearer.

The number of photographs (51.7%) largely exceeds the number of drawings (28.8%) and flow charts (19.4%). There are several reasons to explain this. Firstly, there are more objects than events in the knowledge base. Secondly, when the implementation of EcoLexicon began, quick deployment was considered more important than a coherent view towards image selection. Flow charts and drawings are more reusable than photographs and, if all conceptual propositions in the images are carefully annotated, they can be shown according to the specific perspective of the end-user providing more pertinent and coherent information. The entry for REEF is an example of how this can be done. The three drawings in Figure 21, for example, explain the phases of reef formation as well as the name of the subtype of reef in each phase. The image can be reused for each of the subtypes and, in combination with a colour photograph of the real world entities, will provide the end-user with all the necessary information for comprehension: how each subtype is formed, how it relates to the other subtypes, and what they look like in reality.

Fig. 21 Adequate drawing for reef has_phase fringing reef/barrier reef/atoll and fringing reef/barrier reef/atoll type_of reefSource: http://www.biosbcc.net/ocean/marinesci/04benthon/crform.htm

Flow charts provide much conceptual information and can therefore be used to add new conceptual entries to EcoLexicon. However, this has not been done often. For example, Figure 22 is only found in the entry for UPWELLING, but could have been reused in concept entries such as SURFACE WATER, WIND and CONTINENTAL SHELF. That way, an end-user would be able to understand the interaction among these entities no matter with which one the search started.

24

Fig. 22 Adequate image for conceptual propositions combining UPWELLING, COASTLINE, WIND and SURFACE WATERSource: http://cordellbank.noaa.gov/images/environment/upwelling_470.jpg

With the results of our study and its description of the images, all flow charts can be revised to search for new conceptual information. Moreover, annotating all conceptual propositions in the flow charts in detail as well as their VKPs will boost reusability.

5.2 Correlation between image types, concept types, semantic relations and VKPs

The results of our quantitative analysis of EcoLexicon images (performed only on those found adequate) support our qualitative analysis described in Section 4. According to the results in Figure 23, processes are represented mostly with flow charts, whereas photographs and drawings are used to describe entities. A large number of entities (over 40%) is also represented by flow charts. This is due to the fact that processes can affect physical entities and the latter can cause processes, thus their interaction is better conveyed in flow charts. Processes can also be depicted in a combination of drawings were different phases are shown in each one of them (e.g. Figure 21) and in photographs when the focus is on the result of the process (e.g. Figure 19). Properties (e.g. SOIL PERMEABILITY or ISOTROPIC) appear more often in drawings because you need labels to explicitly convey them, and labels are the most prototypical VKP in drawings (see Figure 26).

25

Photograph Drawing Flow chart0

20

40

60

80

100

120

PropertyProcessEntity

Fig. 23 Correlation between image types and concept types

Not surprisingly, as flow charts are clearly preferred for representing processes, they are also the image type mostly used to convey procedural relations such as result_of and causes (Figure 24; only most representative relations are shown). In turn, photographs are clearly more adequate for type_of and has_location, whereas drawings are used evenly for type_of and has_location, as well as part_of and delimited_by, all typical relations for the description of physical entities. Actually, part_of and delimited_by are specific to drawings.

type_

of

has_locati

on

part_o

f

delimite

d_by

phase_o

f

has_functi

on

result_

of

takes_

place_i

n

made_o

fcau

ses0

20

40

60

80

100

120

140

Flow chartDrawingPhotograph

Fig. 24 Correlation between conceptual relations and image types

Processes are mostly represented in images where relations such as result_of, takes_place_in and causes are present (see Figure 25). Entities, however, are found in images in combination with type_of, has_location and, to a lesser extent, part_of and has_function. This is in consonance with the combinatory potential of concept types in Table 1. Has_function and part_of are less represented because not all entities are functional nor do they have differentiated parts. Moreover, we have found that the

26

has_function relation is not easy to convey in one image (see Figure 20). Properties are mostly present in images that convey type_of and has_location, when describing the properties of an entity. They are also present, approximately in the same percentage, in combination with the other relations, except for made_of, which may be due to the surprising lack of data on this relation.

Fig. 25 Correlation between conceptual relations and concept types

In Figure 26 only the most representative VKPs have been included. All of them are equally present in flow charts in EcoLexicon, they even appear all together in many of them. Photographs have clear referential backgrounds in over 75% of all cases and occasionally include labels. As previously mentioned, drawings are mostly characterized by labels and then arrows. The obvious conclusion we can draw from these results is that flow charts are the most interesting image type for the study of VKPs.

Photograph Drawing Flow chart0

50

100

150

200

250

300

arrows

color coding/contrast

labels

referential background

Fig. 26 Correlation between image types and VKPs

27

0

20

40

60

80

100

120

140

PropertyProcessEntity

6. Image selection and annotation guidelines

This section describes the future extension of graphic information in EcoLexicon and the implementation of specific annotation guidelines based on all our findings described in the sections above.

Image selection guidelines:

1. Use photographs for the type_of, made_of, and has_location relations of physical entities shown in their real-world environment.

2. Use drawings with labels and arrows for representing complex meronymic relations (part_of, delimited_by) or to differentiate between closely related concepts that are otherwise hard to differentiate without making reference to one another. Drawings are mostly fit to represent entities but combinations of several drawings can be used to describe processes and their phases, especially if no flow chart is available.

3. Use flow charts for complex processes and non-hierarchical relations such as causes and result_of. The flow chart must show a high level of referential similarity for the background. It must use colour-contrast to differentiate between closely related concepts. It must also contain arrows to add dynamism and show the direction of the movement or even add textual explanations.

Image annotation guidelines:

1. Annotate image type: photograph, drawing (including maps or diagrams), or flow chart.

2. Annotate all the concepts in EcoLexicon which are present in the image, especially those mentioned in labels.

3. The knowledge base will provide a list with all the possible relations between pairs of the selected concepts. Annotate the most representative propositions for the image.

4. Annotate VKPs: labels, arrows for parts or dynamism, colour coding/contrast, etc.

As for guideline 2, for annotation purposes, the most representative dimensions of a concept can be any type of conceptual proposition, according to the restrictions for each concept type listed in Table 1 or contextual constraints as described in Section 2.

We could benefit from the semantics stored in our TKB in order to allocate images through inheritance, as is done with ImageNet, leveraged by WordNet. For instance, the images of T-SHAPED GROIN will also appear in the concept entry of GROIN. This way, more images would be shown in each concept entry, but if they could be sorted according to the prototypicality or representativeness of the conceptual propositions in

28

the image, no information overload would be caused. In this case, the prototypicality of an image for a concept entry would be based on the number of conceptual propositions present in the definitional template of the concept.

Manual annotation of all EcoLexicon images will be very time-consuming but, as mentioned previously, we believe that the specificity of the graphical information that an environmental TKB requires does not allow for automatic procedures at this point. However, as all newly annotated images will be available to the annotators, the workload will become less in time as image annotation can be refined and duplicating work can be avoided. We believe that it will be worth the effort as the reusability of the images will be greatly enhanced, end-users will be able to view images in the specific context they need, and the internal consistency and coherence of EcoLexicon will improve. In the future, the growing data set that results from on-going image annotation might be used to train automatic annotation systems.

7. Conclusions

Much has been written regarding the importance of combining visual and textual information to enhance knowledge acquisition (Paivio 1971, 1986; Mayer and Anderson 1992). However, the combination of images and text still needs further analysis (Faber 2012; Prieto Velasco 2008; Prieto Velasco and Faber 2012). An in-depth analysis of the features of images provides the means to develop selection criteria for specific representation purposes. The combination of conceptual content, image type based on morphological characteristics, and functional criteria can be used to enhance the selection and annotation of images that explicitly focus on the conceptual propositions that best define concepts in a knowledge base.

Both the multidimensional nature of complex specialized domains, such as the environment, and situational context affect selection criteria. Each change of perspective modifies the way knowledge is represented and thus perceived. The complexity of the conceptual information contained in images related to the environment and the way this information is represented through features such as image type and VKPs, is yet again proof of the multidimensionality of the field. Linking images to concept entries through the annotation of conceptual propositions will provide the means to better represent this multidimensionality without causing excessive information load for the end-user.

Visual knowledge patterns (VKPs) in the images also show dynamism in the sense that they can convey different meanings according to context. Colours, for example, are not only used to achieve a high level of referential similarity, but also to represent contrast within similarity. Arrows, on the other hand, can be used not only to point out the constituent parts of an entity, but also to convey dynamism when representing processes or events.

The analysis of EcoLexicon images has provided the preliminary data to further explore how concept types, conceptual relations, and propositions affect the relation between VKPs, functional criteria and image types chosen for visual knowledge representation. Depending on how the annotation process evolves, new data will provide the clues for future content-based image description and retrieval.

29

Acknowledgements

This research was carried out within the framework of the project RECORD [Knowledge Representation in Dynamic Networks, FFI2011-22397] and the project CONTENT [Cognitive and Neurological Bases for Terminology-enhanced Translation], both funded by the Spanish Ministry of Economy and Competitiveness.

References

Anglin, G., Vaez, H., Cunningham, K. (2004). Visual Representations and Learning: the Role of Static and Animated Graphics. In D. Jonassen (Ed.), Handbook of Research on Educational Communications and Technology (755–794). Hillsdale, NJ: Lawrence Erlbaum.Barriuso, A.,Torralba, A. (2012) Notes on image annotation. arXiv:1210.3448 [cs.CV].Barsalou, L. W. (2003). Situated Simulation in the Human Conceptual System. Language and Cognitive Processes, 18, 513–562.Darian, S. (2001). More than Meets the Eye: the Role of Visuals in Science Textbooks. LSP & Professional Communication, 1(1), 10–36.Di, W., Wah, C., Bhardwaj, A., Piramuthu, R., Sudaresan, N. (2013). Style Finder: Fine-Grained Clothing Style Recognition and Retrieval. Third IEEE International Workshop on Mobile Vision. Portland, OR, USA.Faber, P. (Ed.) (2012). A Cognitive Linguistics View of Terminology and Specialized Language. Berlin/Boston: De Gruyter Mouton.Faber, P., León Araúz, P., Prieto Velasco, J. A., Reimerink, A. (2007). Linking Images and Words: the description of specialized concepts. International Journal of Lexicography, 20(1), 39–65, doi:10.1093/ijl/ecl038.Faber, P., Montero Martínez, S., Castro Prieto, M. R., Senso Ruiz, J., Prieto Velasco, J. A., León Araúz, P., Márquez Linares, C., Vega Expósito, M. (2006). Process-oriented terminology management in the domain of Coastal Engineering. Terminology, 12(2), 189–213.Jeon, J., Lavrenko, V., Manmatha, R. (2003). Automatic Image Annotation and Retrieval Using Cross-media Relevance Models. Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval (119-126). Toronto, Canada.Kageura, K. (1997). Multifaceted/Multidimensional concept systems. In S. E. Wright, G. Budin (Eds.), Handbook of Terminology Management: Basic Aspects of Terminology Management (119–132). Amsterdam/Philadelphia: John Benjamins.León Araúz, P. (2009). Representación multidimensional del conocimiento especializado: el uso de marcos desde la macroestructura hasta la microestructura. PhD Thesis, University of Granada.León Araúz, P., Faber, P. (2010). Natural and contextual constraints for domain-specific relations. In V. Barbu Mititelu, V. Pekar, E. Barbu (Eds), Proceedings of the Workshop Semantic Relations. Theory and Applications (12–17). Valletta.León-Araúz, P., Reimerink, A., Faber, P. (2013). Multidimensional and Multimodal Information in EcoLexicon. In A. Przepiórkowski, M. Piasecki, K. Jassem, P. Fluglewicz (Eds.), Computational Linguistics (143–161). Berlin/Heidelberg: Springer.

30

León Araúz, P., Reimerink, A., Faber, P. (2009). Knowledge Extraction on Multidimensional Concepts: Corpus Pattern Analysis (CPA) and Concordances. In 8ème Conférence Internationale Terminologie et Intelligence Artificielle. Toulouse.Levie, W. H., Lentz, R. (1982). Effects of Text Illustrations: a Review of Research. Educational Communication and Technology Journal, 30, 195–232.Li, J., Wang, J. Z. (2008). Real-Time Computerized Annotation of Pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30 (6), 985–1002.Mayer, R. E., Anderson, R. B. (1992). The instructive animation: helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology, 84(4), 715–726.Mei, T. et al. (2008). Coherent image annotation by learning semantic distance. IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA, 1–8.Monterde Rey, A. M. (2002). Terminología: estudio de las distintas formas de representación conceptual en textos técnicos y su relación con la traducción. In Actas de las II Jornadas de Jóvenes Traductores (147–156). Las Palmas de G. C: Servicio de Publicaciones de la Universidad de Las Palmas de G. C.Montero-Martínez, S., García de Quesada, M. (2004). Designing a corpus-based grammar for pragmatic terminographic definitions. Journal of Pragmatics, 36(2), 265–291.Murphy, M. L. (2003). Semantic Relations and the Lexicon. Cambridge: Cambridge University Press.Nöth, W. (2001).Word and image: intermedial aspects. Medien Pädagogik 6. http://www.medienpaed.com/00-2/noeth1.pdf.Parikh, D. (2013). Visual attributes for enhanced human-machine communication. Allerton, 1126–1127.Park, O., Hopkins, R. (1993). Instructional Conditions for Using Animated Visual Displays: A Review. Instructional Science, 22, 1–24. Paivio, A. (1971). Imagery and Verbal Processes. New York: Holt, Rinehart & Winston. Paivio, A. (1986). Mental Representations: A Dual-Coding Approach. New York: Oxford University Press.Prieto Velasco, J. A. (2008). Información gráfica y grados de especialidad en el discurso científico-técnico: un estudio de corpus. PhD Thesis, University of Granada.Prieto Velasco, J. A., Faber, P. (2012). Graphical Information. In P. Faber (Ed.), A Cognitive Linguistics View of Terminology and Specialized Language (225–248). Berlin/Boston: De Gruyter Mouton.Reimerink, A., García de Quesada, M., Montero-Martínez, S. (2010). Contextual information in terminological knowledge bases: A multimodal approach. Journal of Pragmatics, 42(7): 1928–1950, doi:10.1016/j.pragma.2009.12.008.Reimerink, A., Faber, P. (2009). A Frame-Based Knowledge Base for the Environment. In Proceedings of Towards e-Environment (629–636). Prague.Rieber, L.P. (1994). Computers, Graphics, and Learning. Madison, WI: Brown & Benchmark.Roger, M. (2004). Multidimensionality in Concepts Systems: A Bilingual Textual Perspective. Terminology, 10(2), 215–240.Smeulder, A., Worring, M., Santini, S., Gupta, A., Jain, R. (2000). Content-Based Image Retrieval at the End of the Early Years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12): 1349–1380.Scholz, D. (2000). A solid sense of syntax. Erkenntnis, 52, 199–212.

31

Wenyin, L., Dumais, S., Sun, Y., Zhang, H. J., Czerwinski, M., Field, B. (2001) Semi-automatic image annotation. Proc. of Interact 2001: Conference on Human-Computer Interaction, 326–333.Winston, M. E., Chaffin, R., Herrmann, D. (1987). A taxonomy of part-whole relations. Cognitive Science, 11, 417–444.Zitnick, C. L., Parikh, D. (2013). Bringing semantics into focus using visual abstraction. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3009–3016

32