Visual Design Space Exploration using

Contextual Self-Organizing Maps

Brett Nekolny1, Trevor Richardson

2, and Eliot Winer

3

Iowa State University, Ames, IA, 50011, USA

Self-organizing maps (SOMs) and contextual maps are methods of visualizing high dimensional

data in a low dimensional space. SOMs have previously been applied to visualize characteristics of

optimization problems by generating maps of the component variables to compare interactions and

relationships between design variables. In this paper, SOMs and contextual maps are explored as a

visualization method to directly visualize the design space. Using the techniques described in the

paper, high dimensional datasets are reduced to a 2D, human readable, visual map. Preliminary

results show that the topology of three optimization functions using varying dimensionality can be

clustered and visualized using contextual maps, and information can be gathered from these

clusters including objective function values and variability amongst differing areas of the design

space. This paper focuses on the use of contextual maps to extract valuable information such as

modality and curvature to aid in future work such as selection of appropriate optimization

algorithm and initial point for a solution run.

I.Introduction

isualization of complex design and optimization problems remains a difficult problem due to the inability to

efficiently see data in greater than three dimensions. Accordingly, as optimization problems increase in

dimensionality, the interplay and relationships between variables become harder to discern. In order to comprehend these

problematic relationships, it is necessary to extract the variables and view them independently in one, two, or three

dimensional (1D, 2D, 3D) spaces. Several research projects1,2,3,4

examined various methods to visualize design spaces and

other characteristics of optimization problems. For example, Eddy and Lewis created an environment to view the

optimization process of multidimensional design, but require that the user either view the progress of the performance

objectives, or view design variables in three or less dimensions. Two other alternatives to extracting variables to view them

graphically are either statistical comparison or dimensionality reduction. Dimensionality reduction describes an attempt to

display higher dimensional data by means of low dimensional output, which can be done by discarding variables and

showing only those that describe the strongest correlations, or convert a high dimensional space to a low dimensional space.

Kohonen’s Self-Organizing Map5 (SOM) is a class of artificial neural networks with the ability to reduce dimensionality to a

low dimensional space and furthermore visualize the resulting network. This type of neural network can provide the ability

to train and then inspect a given dataset regardless of dimensionality.

A. Self-Organizing Map

The self-organizing map algorithm utilizes competitive learning strategies to train its (typically) two dimensional

network of neurons. This two dimensional network provides connections between neurons (four immediate neighbors per

neuron in the center) and forms a lattice structure that can be visualized in a perceivable 2D space as shown in Figure 1.

Each neuron has an associated weight vector, as shown in Equation (1), with a dimensionality equivalent to the input

space or data, as shown in Equation (2).

wj = <wj1, wj2, ..., wjk> (1)

x = <x1, x2, ..., xk> (2)

1 Research Assistant, Department of Mechanical Engineering, Human Computer Interaction, Virtual Reality Applications Center, 1620

Howe Hall, Iowa State University, Ames, IA, 50011, USA, Student Member. 2 Research Assistant, Department of Mechanical Engineering, Virtual Reality Applications Center, 1620 Howe Hall, Iowa State

University, Ames, IA, 50011, USA, Student Member. 3 Associate Professor, Department of Mechanical Engineering, Human Computer Interaction, Virtual Reality Applications Center, 1620

Howe Hall, Iowa State University, Ames, IA, 50011, USA, Senior Member.

V

13th AIAA/ISSMO Multidisciplinary Analysis Optimization Conference13 - 15 September 2010, Fort Worth, Texas

AIAA 2010-9326

Copyright © 2010 by Eliot Winer. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.

Figure 1: Diagram of a Self-Organizing Map

6

Equation (1) represents a sample weight vector for one neuron in the lattice, containing a variable number of weights where

k is equal to the dimensionality of the inputs. Equation (2) represents a sample input vector taken from the input space.

The training algorithm for SOMs is a winner-takes-all strategy coupled with a neighborhood influence surrounding the

winning neuron. The winning neuron is discovered by finding the minimum Euclidean distance between the input vector

and each neuron in the map. This method of calculating a winning neuron, and by that a neighborhood of influence, aids in

classifying each new input vector. After a winner is found, a time-varying neighborhood surrounding the winner is updated

using the node update shown in Equation (3).

wj(n+1) = wj(n) + η(n) · hj,i(x)(n) · ( x – wj(n) ) (3)

η is a time-varying learning rate, defining how much the value of each neuron will change throughout the training process,

and hj,i is the neighborhood influence calculated by distance from the winner and time. A fully trained SOM provides a

continuous input space, a spatially discrete output space of neurons, and a nonlinear model of the given dataset7.

B. Contextual Map

Further advances in self-organizing maps provided a modification called the contextual self-organizing map7. Contextual

maps expand upon the trained SOM by generating labels for each neuron in the lattice. The input vectors are fed back into

the map once it is fully trained so that each neuron can acquire a label to describe its weight vector. These labels not only

describe the content of each individual neuron but can also be used to understand the similarity that a neuron has to the

surrounding neurons. Figure 2 shows an example of a contextual generated from research lead by T. Kohonen, where the

letters on nodes are the contextual labels (some nodes have multiple labels).

Figure 2: World Poverty Contextual Map8

Figure 3: World Poverty Map8:

Colors Transferred from Contextual Map

Each label in this map corresponds to a specific country on the world, and the map is grouping these countries based

upon poverty level indicators. The colors displayed within this map were defined by the researchers to connect the nodes

with their countries’ locations on a global map. This way, once the SOM finishes, the resulting colors of each node on the

map can be placed on the world map so that the geographic relationships can be shown. The results of this technique are

shown in Figure 3. This is only one example usage of the contextual SOM. In the case of the research for this paper, a

coloring scheme can be employed to convey additional information about the underlying nature of the dataset. The

principles used to generate this contextual map will be used to gain an understanding and visualization of optimization

problems.

II.Background

The background of this paper describes the current state of research in the area of self-organizing maps applied to

optimization. The authors referenced below have proven that SOMs can be utilized to further understand a design space

visually, and their idea for objective function visualization can be adopted to display the results of this paper.

Su, Shao, and Lee introduced SOM-based optimization (SOMO)9 as a method for solving optimization problems using a

modified self-organizing map. The difference between SOMO and a traditional SOM is the modification made to the

winning node calculation. SOMO’s calculation of the winner uses the objective function, meaning that the 'distance' is the

resulting evaluation of the objective function using the node weight vectors as inputs. This method was also able to plot the

two dimensional lattice in 3D space using the objective function value for each node as the third dimension. The visual

output of the SOMO method proves a viable method for visualizing the design space. The drawbacks of this method are that

it is constrained to continuous optimization problems, and was untested at a dimensionality greater than 30.

Matthews10

used self-organizing maps and applied them to conceptual design of gas turbines and aircraft wings. In both

of these design scenarios the author generated test points from running physical simulations. Matthews was able to use the

component maps of the SOM, which are the extraction of each variable from the trained map, to analyze the design space

interaction. With the extraction of these variables into separate maps, it was possible to inspect the implicit relationships

between variables, which principle component analysis was not able to detect. The author combined the use of SOMs with

the Tanimoto metric10

to automatically extract these implicit relationships between the variables comprising the map. After

the Tanimoto metric had identified potential areas of interest, human intervention was required to verify these as areas of

interest, as the computer only identified potential areas of interest. Overall, the work by Matthews proved to advance the

role of self-organizing maps in design and optimization. This method of examining the component maps could fit in place to

be a next-step after an initial examination of a trained contextual map.

Bishop and Svensen developed the Generative Topographic Mapping (GTM)11

, a modification to the traditional self-

organizing map which maps higher dimensional data to a low dimensional space, similar to the SOM, using latent variables.

The GTM also optimizes its node weight values for the provided dataset, which eliminates the limitations of the SOM in

terms of training parameters, but adds to the computational expense. The GTM was later used by Holden and Keane12

in a

comparison of methods to visualize the aircraft design space, furthermore, this study compared both SOMs and GTMs at

completing the task of visualizing the design space. The SOMs benefited from the ability to analyze individual variables

independently, but the GTM provided an exact solution to the design problem. While the GTM gave a promising result in

solving an eight dimensional problem, the authors noticed a decline in ability to discern a promising solution when

increasing the space to 14 dimensions. It is also important to note that the GTM suffers from increased computation time,

and the inability to extract variables from the design space for evaluation.

Milano, Koumoutsakos, and Schmidhuber13

utilized the Kohonen SOM to reduce the dimensionality and visualize

optimization problems. They were able to train the Kohonen map on the input space of an optimization function following

the rules of the traditional SOM. After the map was trained, they calculated the objective function values associated with

each node's weights. With these objective function values and a trained and organized map the authors were able to draw

contour lines on the two dimensional lattice to show the structure of the input space. The authors also displayed these same

results in three dimensions using the objective function value as the third dimension. This idea follows the same lines as the

visualization completed in the SOMO algorithm in that both methods take the two dimensional lattice and plot it in 3D

space using the objective function value as the third dimension.

The research described above provides a basis for the work in this article, contextual self-organizing maps used to

construct an understanding of the design space, and provide a means to visualize the resulting map. The method described in

this paper will be trained without modifications to the SOM algorithm, and will expand on the ideas presented above. The

first requirement of SOMO and the Milano, et al. methods is the availability of an objective function. Without an objective

function, the SOMO map cannot train and Milano, et al. cannot display the results of their training. Additionally, Matthews

and Holden and Keane require the user to inspect the component graphs of the trained map (variables extracted to show

their pattern), which can be difficult given high dimensionality or very complex problems. The contextual SOM method

described in this paper does not require an objective function, but simply a set of data points. For simplicity in testing, these

points were generated from well-known problems.

This method will plot the resulting lattice map in a two dimensional space to show the topological pattern of the input

space after undergoing dimensionality reduction9,13

. Also, given a visual of the design space, using the idea of extracting the

component variables10,12

can enhance the understanding of the input space by further exploring the interplay between

variables.

This paper attempts to investigate the application of the contextual SOM with optimization functions by:

1. Visualizing the resulting SOM to show the structure of the design space.

2. Verifying the contextual maps display details of the design space by using functions that can be visualized in lower

dimensions.

3. Expanding these initial problems to new problems of higher dimensionality, and unknown design spaces.

Some of the potential results and applications of this work can be tied to the following ideas:

1. Gain information about the characteristics of an optimization problem such as modality (multi-modal), curvature,

linearity, and degree.

2. Provide means for selection of a solution algorithm: numerical, heuristic, etc.

3. Determine distinctiveness of design points and design space, meaning that an initial search area may be found using

contextual maps.

III.Methodology

The algorithm used in this paper follows combined methodologies described above for contextual self-organizing maps.

First, the empty map is generated and then initialized with random weights. Next, the map is provided an initial

neighborhood radius and learning rate, which are standard training parameters. In this case, the initial neighborhood is set to

the entire map, and the learning rate initialized at 0.1 (recommended value from Neural Network7 text) but then decays

exponentially. The map is trained using inputs chosen by randomly selecting points in the design space. The training portion

of this method utilizes the design variable values and the objective function values. An objective function is not necessary to

utilize this method, but was the method by which the data was generated. The training is based upon the winner-takes-all

method as described above where the Euclidean distance determines the winning node. Once the wining node is found, the

update, Equation (3) is used to influence the surrounding nodes in the lattice.

After training is complete, the contextual map phase is executed, which returns the input data to the map, calculates the

winning node for each input data vector, and assigns its associated objective function value to the contextual map label.

Given that the lattice or network contains many fewer nodes than input vectors, each node ends with a number of associated

contextual map labels. Each node’s final label is found by gathering statistical measures to evaluate the node's performance

with respect to the objective function. The first step is to calculate the mean, standard deviation, and minimum value.

Secondly, the data for each node is normalized for coloring purposes described below.

The HSV14

or 'hue saturation value' color scheme was used to display a combination of the mean, standard deviation and

minimum values for the labels of each node. Visually, the mean objective function value can be found by examining the

color of the node, where green is closest and red is farthest from optimal. In other words, red corresponds to a high mean

and green corresponds to a low mean. The standard deviation can be determined by looking at the vibrance of each color.

The more vibrant colors have lower standard deviations and the darker colors have higher standard deviations. The

minimum value is displayed using the whiteness of each node. In essence, the more pigment allowed in a color (further

from white), the lower the minimum value contained within the node. To re-connect with the idea of coloring the map, this

coloring scheme makes it possible to discern the mean, standard deviation, and minimum value of a given neuron and how

its properties relate to other regions of the design space. As an example, a vibrant red node will have a high mean, low

standard deviation, and low minimum value. Alternatively, a dark colored green node will have a low mean, high standard

deviation, and moderate minimum value. Lastly, a white colored node will have a high minimum value. These color

properties provide the ability to compare properties of various nodes within a given map. The understanding of the makeup

of each node allows an investigator to learn more about the design space and optimization problem.

The idea of using contextual self-organizing maps has previously not been applied to the area of optimization, but has

remained in the domain of database organization, and language processing. This paper utilizes contextual maps to explore

the design space in a multivariable visual manner without the necessity of comparing multiple single variable maps to

explain the trends within the design space. Additionally, the HSV color scheme is a beneficial addition to the contextual

SOM because it can display the characteristics of a given node without overwhelming the viewer.

IV.Results

The following presents the results of training the described contextual self-organizing maps on three different functions:

Rosenbrock’s Function15

, Griewank’s Function17

, and Ackley’s Function18

. Data points were generated randomly for each of

these functions at sample sizes (SS) of 1000 and 10000. Each problem was tested with two, five, and ten dimensional

datasets. Each data point generated was accompanied by an evaluation of the objective function for contextual purposes

described in Methodology.

Figure 4 displays the result of plotting the design space of

Rosenbrock’s Valley function. This function was generated and

evaluated as a 2D problem, and plotted in three dimensions with

the vertical axis being the objective function value. Rosenbrock’s

Function (banana function) is a unimodal function with a

parabolic shaped valley that is scalable to n dimensions.

Figure 5 displays contextual SOMs trained to datasets

produced using the Rosenbrock’s Function. Sample sizes (SS) of

1000 and 10000 at 2D, 5D, and 10D are shown. In each case, the

self-organization and contextual labeling reduce the

dimensionality of the visualization to a two dimensional, visible,

space. As described in Methodology, the nodes have set positions

within the lattice, but the containing weight vectors are modified

to best cover the input space during training. This method insures

that the resulting adjacent nodes contain data samples of similar

design variables.

As a proof of concept for the contextual SOM, the

preliminary test cases were 2D functions. This allowed

comparison with observed results of known

optimization problems. One example, the resulting 2D

contextual map in Figure 5A, shows distinctive

properties of Figure 4 (steep edges and low area in the

center). Due to the coloring scheme described in

Methodology, when Figure 4 has a large slope, the

corresponding node areas will contain high standard

deviation causing dark colored nodes. When it has low

mean and is relatively flat, it will result in bright green

nodes. In each 2D case, the map organizes itself in the

same layout as the XY plane in Figure 4 and uses the

described coloring scheme to represent the Z-axis.

This direct representative behavior of the graph can

only be expected for 2D problems because there is no

dimensionality reduction in producing the resulting

map.

Although dimensionality is reduced in both the 5

and 10 dimensional cases, information regarding the

characteristics of the design space can still be retained

from the resulting maps. In Figure 5D, the red

highlighted area is generally a bright green color. As

described in the methodology section, this represents

an area with both low mean and low standard deviation

amongst the data points contained in the nodes. The

edges, which are also in one large connected area,

appear dark brown. This color shows that the mean

value is higher (farther from green), and there is a high

standard deviation (darker color) of the data that each

of the nodes contain. This same trend is generally

observed in each of the remaining subfigures of Figure

5. In each case, this information may be useful to tell a

designer that there is a large area of the given design

space with similar low objective function values, and

there is also a large area ascending away from these low areas into higher values. This information could be enough for the

designer to gain a quick understanding of what the overall design space may look like.

The next function examined was the Griewank Function shown in Figure 6. This function, as in the case of

Rosenbrock’s Valley, is n dimensional. The Griewank function has an overall circular shape filtering to the center, however

has many peaks and valleys and is therefore considered multimodal.

Figure 7 shows that in the 2D training of the Griewank Function, the contextual SOM captures the overall look of the

Figure 4: Rosenbrock's Function16

Figure 5: Rosenbrock’s Function Results

function it has been trained to. It shows both the large

circular curvature toward the center and the sharper

sloping peaks in the corners. These features are

brought out by the large bright green area in the

center as it fades to the outside becoming darker and

getting further away from green. Although the figure

shows the general trends of the data, the 2D scenario

fails to show the multimodal behavior of the graph.

This could be caused by the normalization of the

standard deviation calculated within each node. If

multimodal behavior is captured, the resulting

contextual SOM should show spotted regions of dark

nodes meaning that even though the values are close

in the design space (Euclidean Distance), their

respective objective function values are far apart

causing the node to show color representing a high

standard deviation of the data within said node.

Examples of this observation can be found in

Figure 7(C-F). In the figures, dark nodes scattered

throughout suggest that the data is highly variable

within small regions of the design space. There is no

indication of any large flat or steep regions like the

results of Rosenbrock’s Valley function.

The final function tested was Ackley’s Function shown in Figure 8. This function is also n dimensional. Ackley’s Path

is multimodal as was observed in the case of Griewank’s Function.

Again, the 2D results of the contextual SOM shown in Figure 9 retain the overall shape of the 2D Ackley’s Path function

plotted in Figure 8. Much like the case of the Griewank Function, the 2D results do not capture the multimodal nature of

the function, possibly due to the same reason as before. Results show that there is a large flat area with very high minimum

values (colors close to white), and a small region where the standard deviation increases and the mean objective function

value decreases (colors getting darker and increasingly green).

In the 5D cases (Figure 9C & 9D), results indicate a large flat

plane with high minimum values surrounding a small region with

very high standard deviation sloping toward lower values (shown

by the colors approaching green) similar to the 2D results. Once

again, although the information may be helpful to determine overall

trends, this, as in the 2D test, does not capture the multimodal

behavior of the function.

The scattered regions of dark nodes in the 10D results (Figure

9E & 9F) suggest that the function is multimodal, however, neither

the 1000 or 10000 sample set capture a high plane and small region

to indicate a possible global minima similar to 2D and 5D. This

shows that using different dimensionality can pull out different

characteristics of a given design problem.

Figure 7: Griewank Function Results

Figure 8: Ackley Function18

Figure 6: Griewank Function17

V.Summary and Conclusions

The presented examples in this research are to show the effectiveness of using contextual identifiers in creating a self-

organizing map for use in optimization problems. The results first demonstrate that the contextual self-organizing map can

group similar data points within the design space. Once the data is grouped by the SOM, the map can be colored based on

the statistical properties of each resulting node. With the map colored according to mean, standard deviation, and minimum

value, investigators can gain quick insight into overall behaviors of a given design space.

Future work for this research will include testing larger contextual SOMs (more nodes to train the dataset to), larger

datasets (to train the maps from), and higher dimensionality datasets. Limiting the size of each SOM in the examples

demonstrated may cause a loss of useful information due to the greater reliance on the statistical properties within each

resulting node. Doing so may limit the contextual abilities of the map to represent the overall results. Also, training each

map from a larger dataset may create a more continuous design space resulting in a more understandable, two-dimensional

representation of the design space and its properties. Finally, Solutions to higher dimensional problems are becoming more

and more in demand as the complexity of products continue to increase. Allowing the SOM to organize and the contextual

SOM to visualize a high dimensional dataset could help industry save time and costs by allowing a greater understanding of

the design space in a two dimensional (human readable) format. In conclusion, the immediate goals of our future work are

to gain an understanding of the required map sizes and training parameters that best fit the data sets given certain size and

dimensionality.

Figure 9: Ackley Function Results

VI.References 1 Eddy, J., Lewis, K., “Multidimensional Design Visualization in Multiobjective Optimization”, 9th AIAA/ISSMO Symposium on

Multidisciplinary Analysis and Optimization, Atlanta, GA, AIAA, AIAA-2002-5621. 2 Winer, E.H., Bloebaum, C.L., “Visual Design Steering for Optimization Solution Improvement,” Structural and Multidisciplinary

Optimization,22(3), p. 219-229. 3 Chiu, P-W., Naim, A.M., Lewis, K.E., Bloebaum, C.L., “The hyper-radial visualization method for multi-attribute decision-making

under uncertainty”, International Journal of Product Development, Vol. 9, No. 1-3, 2009, p. 4-31. 4 Stump, G., Yukish, M., Simpson, T.W., “The Advanced Trade Space Visualizer: An Engineering Decision-Making Tool,” 10th

AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Albany, NY, AIAA, AIAA-2004-4568. 5 Kohonen, T., “The self-organizing map”,Neurocomputing, Vol. 21, No. 1-3, November 6, 1998, p 1-6. 6 Koua, E.L., Kraak, M-J., “Geovisualization to support the exploration of large health and demographic survey data”, International

Journal of Health Geographics,Vol. 3, No. 21, June 4, 2004. 7 Haykin, S., “Neural Networks A Comprehensive Foundation”, Prentice Hall Publishing, Second Edition, 1999, ISBN: 978-

0132733502. 8 Kaski, S., “Example Application of the SOM: World Poverty Map”, Neural Networks Research Centre, September 1997.

[http://www.cis.hut.fi/research/som-research/worldmap.html. Accessed 2/12/2010.] 9 Su, M.C., Zhao, Y.X., Lee, J., “Som-based Optimization”, IEEE International Conference on Neural Networks, 2004, 781-786. 10 Matthews, P.C. “The Application of Self Organizing Maps in Conceptual Design”, PhD thesis, Cambridge University, Engineering

Department, 2001. 11 Bishop, C.M., Svensen, M., Williams, C.K.I., “GTM: The generative topographic mapping”, Neural Computation MIT Press, Vol.

10, No. 1, 1998, p 215. 12 Holden, C.M.E., Keane, A.J., “Visualization Methodologies in Aircraft Design”, Proceedings of the 10th AIAA/ISSMO

Multidisciplinary Analysis and Optimization Conference, Albany, NY, 2004, p 1-13. 13 Milano, M.. Koumoutsakos, P., Schmidhuber, J., “Self-organizing nets for optimization”, IEEE Transactions on Neural Networks,

Vol 15, No. 3, 2004, p. 756-758. 14 Clark, A.C, Wiebe, E.N., “Color Principles – Hue, Saturation, and Value”, NC State University, May 2000,

[http://www.ncsu.edu/scivis/lessons/colormodels/color_models2.html. Accessed 2/12/2010.] 15 Hedar, A., “Test Functions for Unconstrained Global Optimization – Rosenbrock Function”, Global Optimization Methods and

Codes, [http://www-optima.amp.i.kyoto-u.ac.jp/member/student/hedar/Hedar_files/TestGO_files/Page364.htm. Accessed 8/4/2010.] 16 “Rosenbrock’s function”, MedWiki, [http://medlibrary.org/medwiki/Rosenbrock's_function. Accessed 8/24/2010.]

17 Hedar, A., “Test Functions for Unconstrained Global Optimization – Griewank Function”, Global Optimization Methods and

Codes, [http://www-optima.amp.i.kyoto-u.ac.jp/member/student/hedar/Hedar_files/TestGO_files/Page364.htm. Accessed 8/4/2010.] 18 Hedar, A., “Test Functions for Unconstrained Global Optimization – Ackley Function”, Global Optimization Methods and Codes,

[http://www-optima.amp.i.kyoto-u.ac.jp/member/student/hedar/Hedar_files/TestGO_files/Page364.htm. Accessed 8/4/2010.]