The articulation of visual experiences through algorithm

the articulation of visual experiences through algorithm

Carlos Lunetta

The articulation of visual experiences through algorithm

by Carlos Lunetta

Thesis submitted in partial fulfi llment of the requirements

for the degree of Master of Fine Arts and is approved by the

MFA Design Review Board of the Massachusetts College of

Art in Boston.

May 2005

Jan Kubasiewicz

Thesis Advisor

Joe Quackenbush

Thesis Document Advisor

Brian Lucid

DMI Faculty

Toby Bottorf

External Reviewer

Abstract

The Articulation of Visual Experiences Through Algorithm explores the

concepts and possibilities computation presents as a creative medium for

design.

This thesis investigates the relationship between code and visual language.

From this relationship, a new vocabulary of new media will emerge,

driven by ideas that stretch their existence simultaneously through all

means – visual, aural, verbal, haptical, temporal and logical.

My methodology was based on discrete experiments native to the compu-

tational medium which investigated a range of topics including: musical

intervals; generative and interactive form systems; interactive op art;

database visualization; and the plastic components of time.

7

9

Aknowledgements

I traveled 4817 miles from São Paulo to Boston to improve myself as a

designer. This would have been impossible without the support of my

family, but I own absolutely everything to my wife Margarida, and her

love, support and patience.

I’d like to express my gratitude to Jan Kubasiewicz, not only my mentor,

but also a good friend.

To Joe, Gunta and Brian, for all the worthy hard work.

To my fellow classmates, Stephen, Keiko, Kate, Karolina and specially

Lynn (and Ronaldo) for all the support and friendship.

To my friends Colin Booth and BJ Jones, for trusting and receiving a

Brazilian weird student at the US.

To Dave and everyone at Beam interactive, for the opportunity of

practicing the thesis preach.

To Ben Fry, Mario Klingemann and all Processing community.

And fi nally to Google, for returning 264,000 thesis acknowledgments to

my amusement. There are surely a lot of grateful people out there!

11

Table of Contents

Defi nitions 17

Roots 27

Concepts 39

Praxis 53

Experience 67

Case Studies 71

Sound in Space 73

Form Systems 85

Riley Patterns 99

Logic Molecules 109

Patterns of Time 115

Findings 121

Bibliography 125

13

Preface

“Every work of art is the child of its age [...] Each period of culture

produces an art of its own which can never be repeated.”

— Wassily Kandinsky

15

Disclaimer

“To defi ne is to limit.”

— Oscar Wilde, in “The Picture of Dorian Gray”

The limitation serves a purpose - better understanding.

This thesis document defi nes a series of concepts and ideas, many of

them vast, open and entangled. By any means those defi nitions intend

to limit concepts – they mean to raise understanding by disclosing

relevant issues.

Ideas are not – and shouldn’t be - limited to their verbal counterparts.

17

Defi nitions

Visual experience, defi ned.

Design is an act of assembly of form through composition.1

A composition that has a purpose: to communicate.

The idea of communication implies that a message will eventually be

received by someone; there is an audience, a receptor. Also, communi-

cation implies the existence of a medium, a dimension where composition

becomes reality.

Composition, audience and medium are linked together: each one infl u-

ences and is infl uenced by the other, in a mutual, constant change.

The term “visual experience” is defi ned by the connection of these three

concepts; a composition acknowledged of its audience and aware to the

nature of its medium.

��

��

��

��

1 “Design is the method of putting form and content together. Design, just as art, has multiple defi nitions,

there is no single defi nition. Design can be art. Design can be aesthetics. Design is so simple, that’s why it is

so complicated.” – Paul Randso complicated.” – Paul Randso complicated.”

a model for visual experience

18

Computer: a new medium

The personal computer has worked as a tool for design production and

artistic experimentation since its tender age. During the mid-nineties,

though, a shift in the role of computing occurred: the advent of the

internet, that allied with the proliferation of household computers gave

birth to a new mass communication medium.

Computational devices2 became a medium, with an important difference

from other known media – they allow direct feedback and participation

from the audience. The concepts of broadcaster and viewer got blurred,

since one can become the other after few keystrokes.

As a medium, computation affects the audience. Consequently, graphic

design is infl uenced by the change; the new medium opens a myriad of

possibilities for creation in time, space, shape and sound. The partici-

pation of the audience raises the concept of interactivity.

Computers have became no longer just a production tool, but also the

living space for the design creation.

New media creativity

Once a new medium has been established and the gates for new dimen-

sions in design have been opened, a question arises: how is the conception

of visual experiences changed? How can new media affect creativity?

The initial transformation in design was an adaptation; new media design

was a collection of items from different media bonded together. A print

design piece would receive navigation buttons, a small animation intro

and be published as a “new media” piece.

The adaptation evolved according to the features that mainstream

graphic software packages would offer; software companies like Adobe

and Macromedia ended up dictating the aesthetics of interactive com-

munications by offering “canned” solutions, like templates and plug-ins,

2 Although this document aims the exploration of personal computers, like desktop or portable systems, any

micro processed object is considered a computational medium: cell phones, palm tops, text messengers, DVRs

and others

19

and generating design elements according to their own proprietary algo-

rithms, like vector antialiasing and motion tween functions.

Up to this point in time, the creative process per se changed very little

– more dimensions were added, but the visual experience conception

remained the same. Under the navigation buttons and animated items,

conceptually print media pieces live online.

Birth

The computer is both a medium and a tool for design – but the creative

process seems to still be far from the machine. Why is it the case that the

principles of computation are not present in the creative framework, in

the conception of a visual experience?

They are present, but for a very small number of designers, still far from

the mainstream. There’s a gigantic gap between the number of designers

that use computers as a tool and the number of designers that use the

computer as a part of the concept in the design work. There are a small

number of design pieces that can only exist and be understood inside the

digital medium.

The language of design is in constant evolution, and now it faces the

opportunity to embrace the concepts and processes of its new hosting

medium, becoming native, being reborn.

What are the concepts of computation?

Computation is about change.

The personal computer is a highly evolved calculator, performing billions

of arithmetical and logical operations per second. The processor works by

getting certain inputs, performing calculations and outputting them; data

is constantly being chopped to small instructions, calculated and rebuilt.

Let’s use a keyboard input, for example; the physical keystroke is

converted to a binary signal; the signal is processed and converted to a

visual output, the graphical representation of the key – that is displayed

20

on screen. The core of the computational process is about translating,

converting a certain input to a certain output.

The creative use of computation takes advantage of the nature of data pro-

cessing in favor of design. There are situations where traditional design

meets the limits of human labor; a computer can address a different scale

than human hands in artistic creation, dealing with the very large, the

very small, the very precise, the very exhausting, so on and so forth.

Some of the computation characteristics are:

• Transformation: as stated above, the main concept of compu-

tation is to transform data.

• Repetition: a processing instruction can repeat itself eternally, as

long as its physical part works. An element can repeat itself over time

or over space – the replication is lossless.

• Randomness: true randomness comes from nature, in events

like radioactive decay. Computer systems feature pseudo-random

numbers, a formula that produces no repetition, smooth numeric

distribution, and more important: the lack of predictability.

• Precision: computer processing elements can be extremely exact

and precise when necessary.

• Predictability: once a set of rules is created, the code behaves

accordingly. Just like the physical world is attached to the laws of

physics, a program can be attached to a set of unbreakable laws that

will be always obeyed.

a model for computation

��

��

��

��

��

��

��

��

��

��

��

��

��

��

21

• Discretion: program elements can be discrete; events can be

triggered by any kind of input rather than linear time. This is the key

for interaction.

• Complexity: computation can achieve and display unbearable

amounts of visual or logical information.

• Simplicity: the opposite of complexity; the ability to complete a

task with a clear, austere and minimal output.

• Remembrance: the ability to store data, and recall it later, with no

logical capping of time or quantity – the only limit is physical.

The concepts of computation are implemented in a computer code as

formulas, or sets of instructions – known as algorithms.

The origins of the gap

There is a resistance in incorporating computer processing characteristics

in the concepts of design. The resistance has mainly two reasons;

The fi rst reason is the diffi culty in communication between humans and

computers; the computer language is different than common verbal

language. Some specifi c training is necessary so one can be fully able to

read and write computer code. At the same time, the amount of training

today is a small fraction from the amount of training twenty years

ago. Computer language is evolving at a similar speed of the computer

hardware evolution. Coding is more accessible now than ever, and there’s

a prognostic for future improvements. The challenge is to learn what to

tell the computer – and not simply how to tell it. Language syntax depends

on the available technology, but the idea to be coded is timeless.

The second reason is cultural and social; western education branches arts

and humanities from math and science. From high school to postgraduate

education, there’s a strong sense of separation between different areas

of knowledge. Computational design merges art, math, physics and even

psychology, when dealing with the study of interactive design reaction.

The union of different branches of knowledge is required – accompanied

by a mindset prepared to create blissfully in any necessary fi eld of study.

22

Verbal, visual and computational

How different is the computational language from visual language?

The human skill of communication is granted by sensorial channels;

ideas, facts and feelings are understood and expressed whether in verbal,

visual or any other sense.

The compounding elements of a channel have similar counterparts on

another channel; correlations can be addressed between elements with

similar conceptual functions

The correlation of elements has a clear role in the understanding of

language; think about the word “red”. This word has strong visual in-

formation attached to it, representing the visual hue known as red. The

experience of red can be expressed verbally, through the word, or visually,

through the hue. It also can be learned both ways.

We can understand a concept by contextualizing it in a communication

channel; words can describe visual objects, phrases can describe visual

actions - and vice versa.

��

��

��

��

��

correlations between verbal and visual domains

23

The computational language acts as an extra channel - it has similar

structural properties to other communication means, thus, correlations

of similar elements in the structure can be established.

The computer language, in essence a binary series of logical and arith-

metical operations, can be understood by contextualizing other channels

– especially, but not limited to, verbal and visual realms. We can commu-

nicate to a computer by typing verbal commands; the verbal will become

a digital process, returned and converted to any output – verbal, visual,

haptical, sonorous or any other available.

Language is a system4, where elements are composed to form meaning;

The structure of the computational language is similar to the structure of

visual, verbal or any other language; a structure of elements interacting

in a composition. Elements can be arranged to work together in a logical

environment just as in the visual or verbal form.

��

��

��

��

��

4 “A language is a system of pure values” – Ferdinand Saussure“A language is a system of pure values” – Ferdinand Saussure“A language is a system of pure values”

domain correlation examples

24

��

��

��

��

��

��

��

��

��

��

��

��

��

��

correlations between verbal, visual and logical domains

��

��

��

��

��

��

��

��

domain correlation examples

25

Module and System

Module is an independent part, a single element that exists by itself,

having built-in functionalities. System is the working interrelation of

modules, comprised as a whole.

Scale is a determining factor to defi ne something as a module or a system;

an atom can be taxonomized as a module – a unique, independent part -

or a system, formed by the particles and sub particles that interact within

its apparent unity.

The building blocks that defi ne a language are considered modules; each

module is unique, and represents a concept, an idea. Modules interrelate,

creating a system - the composition. Words interact in a sentence, shapes

interact in a canvas, and programming objects interact in a code.

Articulation

The articulation of ideas in the visual, verbal or computational dimen-

sions is based on the composition of elements.

As initially stated, design is composition - acknowledged by its audience

and medium. The articulation of visual experiences can grow along the

creative potentials of the new computational medium;

It’s time to articulate visual experiences through algorithm.

��

��

��

��

�� the structure of model/system, with an example of application.

27

Roots

Art, Science and Religion

The underlining concepts of computation and design have met many

times through history; accessible computation may be a new format, but

mathematical and scientifi c concepts aligned with art are not.

The branches of knowledge have been treated as one through time; the

knowledge of understanding nature, the knowledge of understanding

the spirit and the knowledge of expressing both. Science, religion and art

are linked together in the history of civilization. Pyramids, temples and

statues that remain still prove how engineering and art were combined in

service of belief.

Notable thinkers in history also show that the path of science and art can

be one; Greek philosophers like Plato and Aristotle, for example, inves-

tigated math and physics along with aesthetics and poetry – all inquiries

about the real world.

Jumping in time and space, one of the best examples is Leonardo da Vinci.

Da Vinci had a strong grasp of mathematics and engineering, showed

in his fabulous inventions, and strong artistic ability, proofed with his

invaluable paintings and sculptures. His creativity had no borders.

a solid articulation of geometry.

28

The renaissance era of da Vinci gives us an important demonstration on

the incorporation of mathematical and scientifi c concepts in the artistic

creation. Before this movement, medieval paintings were visually fl at.

Space and volume were represented, but did not refl ect reality. The artistic

composition, as a concept, was focused in the symbolical components;

the concept of composition was centered in the meaning.

The renaissance marks the advent of perspective techniques. Perspective

drawing was initially a work of architects only, used in construction

projects. But the new aesthetic of space spread and many talented painters

embraced it when creating their work.

“Art as opposite of science? Masaccio, who is often hailed as the fi rst painter of modern times, was a member of the guild of apothecaries and grocers”

— Karl Gerstner

In a non-digital way, Renaissance paintings are visual experiences ar-

ticulated through algorithm; the formulas of perspective were carefully

implemented when picturing space, scientifi c data was used to create

visual compositions and geometry became part of the creative process.

Sassetta’s The Meeting of Saint Anthony and Saint Paul:the vanished vanishing point

29

Advancing more in time, we arrive at the modern model of knowledge

categorization; each fi eld of study is highly defi ned and specialized. As

civilization evolves, science grows in complexity – it’s intangible to grasp

all fi elds of knowledge. Art became a distant branch, and even the lower

stages of education impose a learning path choice.

“After a certain high level of technical skill is achieved, science and art tend to coalesce in esthetics, plasticity, and form. The greatest scientists are always artists as well.”

“All religions, arts and sciences are branches of the same tree”

— Albert Einstein

Masaccio’s Trinity. Masaccio employed complex mathematical detailing when producing his work.

30

Physicists like Albert Einstein and Werner Heisenberg believed that

knowledge has similarities in different specializations – and there’s a

mutual benefi t of connecting those similarities.

The Massachusetts Institute of Technology (MIT) plays an interesting role

in this relationship. Renowned for the development of groundbreaking

technologies in different fi elds of science, MIT maintains the CAVS

– Center for Advanced Visual Studies, founded by Gyorgy Kepes. Kepes

was a designer that did not believe in the separation of art and science;

his explorations on visual language and his books refl ect this idea, and

for almost thirty years (1946-74) his presence at the MIT maintained this

ideal alive.

“An experimental effort to encourage such interthinking between different disciplines in the visual arts and scientifi c and technical fi elds is more than overdue. As the twentieth century has grown older, such intercommunication has become seemingly more improbable”

“The dynamic interdependence of the wisdom of the senses and wisdom of reason is fundamental to a balanced growth”

“Yet, history shows us that art and science, two basic human activities springing from a common love of nature are interde-pendent. Each achieves stronger growth when cross-pollinated by the other.”

— Gyorgy Kepes

Computation

The domains of visual and computational languages met right after the

advent of processing machines, during the fi fties. The early explorations

involved tweaking the graphical output of huge computers to record fi lm

frames and create animations, or by simply photographing an oscillo-

scope display that could be programmed. John Whitney, Sr. was one of

the most important founding fathers of computer graphics.

31

“The union of color and tone is a very special gift of computer technologies”

“Time has become visual”

“My computer program is like a piano. I could continue to use it creatively all my life.”

— John Whitney, Sr.

These quotes regard Whitney’s explorations on computational abstract

animation and music1.

In 1962, A. Michael Noll at New Jersey’s Bell Labs created a computer

program that featured an algorithm whose function was to build visual

compositions; the reference for the algorithm rules was the composition

style of Mondrian.

The work of Noll has extreme importance; besides Mondrian’s mimic, his

programs investigated composition from a probability stand point – visual

forms were derived from a computational concept. Lines would be drawn

(or not) according to the visual density of the composition. Other Noll

Oscilloscope recording by Whitman

1 Two of his short movies can be accessed at

http://www.siggraph.org/artdesign/profi le/whitney/nobio_intro.html

32

works dealt with geometric randomness or repetition; he clearly took

advantage of the newborn computation process in a creative way. The

machinery evolved a lot since then, but Noll’s ideas are still fresh – there

is no conceptual obsolescence.

At the same time, Douglas Engelbart was creating the foundations of

electronic communications and computational offi ce work. Engelbart

developed the mouse, the hyperlink, video conference, fi le structure

navigation, word processing and more. Although he wasn’t pursuing

artistic objectives, his work supports the employment of creativity in the

computer medium, proposing to use the machine to augment the human

intellect.

The MIT made strong contributions in the graphic computing fi eld. In

1963 the Sketchpad was created – the fi rst CAD (Computer Aided Design)

interface. Sketchpad was extremely advanced for its time; the user would

interact with the application directly on screen, with a pen, through a

graphic interface. All the interaction concepts of this system are the

basically the same we use today.

In the early seventies, Xerox Corporation founded the Palo Alto Research

Center, also known as PARC. It was inside the PARC that most of the

“modern” concepts used in computation were invented, like the icon,

object oriented programming, laser printer, WYSIWYG (what you see

Noll’s Mondrian experiment

33

is what you get), computer notebooks, paint and animation applications

(before that, everything had to be coded or use dedicated hardware), midi

composition and many other ideas that shaped the terms “multimedia”

and “graphical interface”. The PARC computers operating system was

called SmallTalk and it featured something that is still lacking in today’s

operating systems: seamless integration between the graphic interface

and the code behind it. The user was able to customize, change and create

new functions whenever it was necessary, since there was no barrier to

explore the operating source code. PARC successfully conducted experi-

ments where teenagers were taught to code in SmallTalk language, and

soon they would use their creativity to generate graphic applications.

SmallTalk can be considered the golden age of computational design,

when the visual and the logical sides were always connected and evident.

The contemporary contribution from the MIT comes from a new gen-

eration of scientists/artists/designers developing projects regarding the

The SketchPad

PARC SmallTalk - 1976

34

interaction of humans and technology. This happened at the “Visual

Language Workshop”, in 1973, and later, in 1982 when the Media Lab was

founded by Nicholas Negroponte. The specifi c topic of visual language

mixed with computation was more intensely researched years later by

Muriel Cooper, a pioneer in digital typography who worked to develop the

language and structure of electronic design, and to expand the graphic

design culture to four dimensions. John Maeda, considered Cooper’s

successor, founded the “Aesthetics + Computation”2 group, where many

researchers were involved in visual issues through programming, and how

the visual language could be expressed and interacted with an audience

through computation.

The Aesthetics and Computation group started its operations at the

same period that the internet blossomed and desktop computers became

present in common households. Designers started to create for the

computational medium; the graphic software packages would keep the

programming language underneath a friendly interface, “protecting”

designers from the necessary internet coding. The edge of programming

for artists was consolidated with Macromedia Flash and Macromedia

Director, commercial software packages where advanced users could

script their own visual behaviors code.

The term “Aesthetics + Computation” wasn’t just university lab concept

anymore – it was present in the society. One of the projects from Maeda

dealt with accessible computer coding focused in visual elements, called

“Design by Numbers” – a coding interface whose strength was in its

minimalism, where the communication between designer and machine

could be clear. Maeda’s project appears to have strong infl uence from

the previous “Turtle” project from MIT, a software application where an

element on screen called turtle could be moved through code in order to

build images. The turtle project later was know as Logo, and was largely

used in schools during late seventies/early eighties. Logo helped to teach

the basic concepts of computing – the same idea that Design by Numbers

explore. John Maeda’s work also helped to spread the idea that design

could be achieved through computational means, that creativity could be

expressed through code with his interesting explorations of algorithmic

visuality.

2 http://acg.media.mit.edu/

35

The second project from MIT towards a visual creation code was

created by Ben Fry and Casey Reas, and is called “Processing”. It adds

a great number of features and extend “Design by Numbers”, providing

more creative freedom and keeping the accessibility for non-computer

scientists. Processing has today around 1000 users – a small number

compared to the whole interactive design community, but a big number

of participants in a non-commercial, non-advertised university project,

that is still entering its beta stage of development. The most important

fact: the massive majority of the users are interactive designers, who were

initiated to scripting languages when developing for the internet, and

sought ways for doing more. Processing has no timeline, no visual thrills,

and its interface is a simple text editor. But the concept of the application

is clearly targeted to the artistic creation, hence its success among the

design community.

Logo interface

Maeda’s Design by Numbers interfaceNumbers interfaceNumbers

36

Processing Beta interface

39

Concepts

Creative Framework

The idea of a creative framework can be explained through examples; let’s

use painting and photography. Two forms of expression that share similar

characteristics – they’re visual, compositive, bi-dimensional, have color

hues or tones that form shapes and they communicate through visual

information.

Although similar, they’re also very different. It’s easy to discern one from

another; they have different aesthetic values, and generate completely

different reactions.

Photography strongly refl ects reality, augmenting or diminishing an

aspect of it by light, texture and positioning; photography is subtractive

– the artist takes the physical world and frames a theme out of it, a part

out of the whole. The kinds of camera, lenses, time of the day or season of

year are manipulable elements that the photographer can use to achieve

his intended visual result.

Painting has its strength on expressing the author’s internal world; even

a photo-realist painting translates the artist’s view of light, color, form.

A painting is built from a clear canvas; it is additive, colors and shapes

are added to build a visual piece. Pigments, solvents, brushes and stroke

techniques are elements that the painter can manipulate on creating his

work.

Both examples - painting and photography - are visual experiences; com-

positions that communicate in the visual realm. But they’re built through

different creative concepts, using different creative tools.

Creative concepts are the mechanics of expression in a medium – the ways

that an expression can become reality. Light is a concept in photography;

space is also a concept. Both determine how the visual experience of a

shot will be. In the same way, shapes, colors and dimensions are concepts

40

in painting; each medium has its own set of procedures. The instruments

that allow those concepts to materialize - lenses, brushes, pigments, fi lm

– are creative tools.

Computational Concepts, revisited

As stated initially, computation is about change - and its mechanics feature

concepts like transformation, repetition, randomness and others.

Pure computation, just like pure mathematics, is an abstraction; it does

not necessarily have a component in the visual domain in order to exist.

If a piece of code is written on screen, the written letters are just a visual

representation. The computational language can be reduced to a series

of electrical signals traveling trough logic circuits. The elemental state of

computation is a traveling electron.

It might appear diffi cult to correlate such abstract medium to the creation

of visual experiences, but we can compare it to cinema; the nucleus of the

moving picture is time –the articulation of visual experiences through

time. What is the visual component of pure time? Can time be touched,

seen? Time is an abstraction that applied to the photographic medium

produces montage – composition in time, a movie.

We are able to understand pure abstraction by contextualizing it in a

tangible medium. Film, for example, contextualizes time through image

montage - fl ashbacks, slow motions, time lapses - the fi lm strip is able

to demonstrate properties of time visually. The pure abstraction of

computation can be contextualized - visually, through pixels, or verbally,

through code. The intangible “traveling electron” can, fi nally, generate a

visual experience.

41

Modules

Scale determines if a concept should be understood as a module or as a

system – the category depends on the point of view. In a standard user

level point of view, the general computational module is called data.

Data is a computational fact, the declared digital information. The

category of data is called data type. There are two kinds of data types:

primitive and composite.

Primitive data is single information, attached to a value. Numbers are

primitive data. There several categories of numbers, according to the

amount of bytes they take into the computer memory. Logical states (true

or false) and single characters (“a”, “b”, “c”) are also primitive data. The

value of a declared data can be fi xed, or vary according to a function; data

with mutating values is simply called variable.

��

“a”

��

��

11.0

1e201.77e-3

truefalse

“hello”

{1, 2, 3, 4}

class example{ var example1;var example2; }

��

��

��

��

��

��

��

��

42

Composite data is the name given to primitive data combined; a word,

for example, is a combination of characters, and is called string. Arrays

are another kind of composite data: they’re groups. They can be groups

of other groups (nested arrays) number, characters, objects or any other

datatype.

Classes and Objects are composite data. They’re an important concept

to be grasped, since almost all modern computation is based on Object

Oriented Programming. Object is a philosophical term: it defi nes

something, a particular item that has properties and relations. A software

object can be compared to a physical world object, or a visual object.

Every object belongs to a class: the class is the template, the category that

defi nes the possible properties that an object can have.

> this.teaPot

a beautiful physical object a reference for a logical object

things the make a real teapot be a teapot:

1.ability to hold liquids2. handle3. lid4. spout

class Teapot {fl oat size;fl oat color;String lidString spout}

class “Teapot”: things the make a teapot object be a teapot.The initial defi ned properties are size, color, the kind of lid and the kind of spout.

43

The main feature of programming through objects is the modularity:

objects can be multiplied, reused, changed, and are independent beings

inside a system.

Another important computational concept is the function. A function

describes an action to be taken, the relation between elements to be

realized; it drives the change on any kind of computational module.

Modules in action: system

Computation is about change; through functions or other methods,

computational elements can be put to work, perform a concept – act as

a system.

Let’s review some of the systematical concepts of computation:

• Repetition

A system is never truly static – except when data is stored in a physical

media, its existence lies within electrons. A static module is an illusion: it

is a module that is repeating itself over time, reaffi rming its existence on

each circuit cycle.

The invisible repetition can be visual, when a graphic element stands

in time - or repeats itself over space. The simultaneous repeated copy is

called instance.

In the following example called repetition.pde, coded using Processing,

a rectangle object is repeated over space - and infi nitely, over time. The

code also performs two iteration loops to proper position the instances

along a grid.

void setup() { size(100,100); fi ll(0);}void draw() { background(255); for(int i=0; i<3; i++){ for(int j=0; j<2; j++){ rect(10+i*30, 25+j*30, 20, 20); } }}

44

• Transformation

When a property of a module is being changed over space, or time, or

any kind of input, a transformation is occurring; almost everything in

computation is an example of transformation.

The following simple code called transformation.pde increases the value

of the x coordinate of a square over time. When the edge of the screen is

reached, the value is reset, and the transformation starts again.

• Randomness

As initially written, true randomness comes from uncertain events in

nature. A known source of uncertainty is the radioactive decay, or the

movement of gas atoms in vacuum. Randomness is a controversial topic

in physics and mathematics, but no technical background is required to

use the idea of unpredictability in while designing. Random algorithms

produce a number inside a certain range; the range can be a space

coordinate, size, or a moment in time. Any property can be tweaked in

random ways.

The following example (randomness.pde) generates a random coordinate

for the square each frame:

int x = 0; // coordinate in x axisvoid setup() { size(100,100); fi ll(0);}void draw() { background(255); x = x+1; if (x>=width) { x = 0; } rect(x, 40, 20, 20);}

fl oat x; // coordinate in x axisfl oat y; // coordinate in x axisvoid setup() { size(100,100); fi ll(0); }void draw() { background(255); x = random(width); y = random(height); rect(x, y, 20, 20); }

45

• Precision

The generation of images through algorithm sometimes requires a great

deal of mathematical processing; certain values go through hundreds,

thousands of calculations – and the visual output depends on how exact

those calculations are perfomed.

The case study of the harmonograph (page 78), for example, shows

musical intervals converted to images; the purity of a interval determines

if the image will be symmetrical and stable or not. The visuality of the

whole piece changes with any small deviation of the interval value, even a

0.0000001 unit. The concept of precision thus is valuable when obtaining

certain visual pieces.

• Predictability

Predictability comes from nature; for example, it’s absolutely predictable

that all physical objects on Earth will obey the laws of gravity. Physical,

chemical and biological processes occur with confi dence that they were

supposed to happen. In a computation system this concept can be

recreated; a function can set behaviors to any module, and this behavior

will happen, always, while the system is active. Creatively, it allows the

designer to create his own rules: like that all objects should vibrate all

the time. Or everything should become red when touched and so forth.

Computer bugs are the outlaws of predictability.

harmonograph: any change in the interval value creates a different visual representation

46

• Discretion

Being modular and being discrete are almost synonyms; events in com-

putation don’t need to be linear; events can happen in modules – they can

be handled through any kind of input. A computer code can have a set of

functions, and these functions be called to act any time. A user interacting

with a mouse is an example; when the mouse clicks an item, an event is

called.

The mouse was the event that determined the moment in time where

the action took place; that action could happen again and again – or not.

There’s no need for linear time in logical modules.

• Complexity

The concept of complexity is to use computation to achieve a degree of

element profusion and element interaction that is intangible to human

hands; based on the principles of repetition and transformation, com-

plexity can become more an aesthetical statement, searching beauty in

the intricate.

• Simplicity

Simplicity is the holy grail of computation, since the complexity is a

natural tendency; bulk processing powers opens extensive creative

options. A visual experience can be successful either from complexity or

simplicity.

Peter de Jong algorithm shown with 2,000,000 iterations.

47

• Memory

There are two kinds of memory in a computer system; the volatile, logical

memory – where the necessary data for realizing a code is stored, and the

physical storage.

The logical memory has a temporary nature; to perform operations, the

computer stores data of any datatype in its functional memory banks.

A computer can retrieve with precision any item from its electronic

memory.

The physical memory is permanent1; the role of time is very clear on the

concepts of computation, and the physical memory is the place where

time stands still. Like a printed book, binary data is printed onto an

optical or magnetic surface, where it can be physically stored.

The creative value of storage is in its power: a database can hold an un-

imaginable amount of information, in any kind of format. The internet

itself is a gigantic database, that can be retrievable through search engines.

The Human Genome Project and the CERN nuclear collision database are

examples of huge unique data storage projects, with billions of bytes that

can be retrieved, sorted and shaped in any way.

Malevich’s Black Square and Red Square : simple and powerful.

1 Subject to physical decay through time; under controlled storage conditions, the decay is minimal.

48

The concepts of computation can be visualized, or verbalized, in countless

forms; the key to achieve visual experiences through them is to employ

artistic creativity in all imaginable – or unimaginable - ways.

The issue of generative design

Generative design is a term vastly employed to describe visual pieces

generated through computational processes; but its true concept is not

clear. Between its diverse defi nitions, it can be used as “design generated

by a computer” or “design generated through a computer” – very different

ideas.

Can a computer design?

Algorithms can generate visual compositions from ether; visual elements

can be built and tweaked in any amount, following all the concepts of

computation. The illusion that a computer is designing comes from the

capacity of variation and randomness on the construction of “new” com-

positions.

Sadly, random and abstract are not synonyms. Abstraction is a thought

process. Computers don’t think (at least yet).

Ben Fry’s Haplotype Lines: a creative visualization of the massive genomic database.

49

The computer intelligence is a widely discussed topic; one of the most

famous writings on it comes from Alan Turing, a mathematician that

published in 1950 a test where machines would be considered intelligent

if they could engage a non discernable conversation with a human being.

The validity of the Turing test is questioned today, but no machine could

achieve the “victory” so far. Despite the advancements on artifi cial intel-

ligence, it remains artifi cial - a mimic, or a source of algorithmic varia-artifi cial - a mimic, or a source of algorithmic varia-artifi cial

tions.

The conception of visual experiences is lead by original ideas; algorithms

can be programmed to follow the principles of visual language – pro-

portion, distribution, color harmony; they also can be programmed to

follow style guidelines – use certain color harmonies, certain line styles,

and achieve an overall aesthetic sense. But the leading idea remains in the

hands of the designer.

The term generative design shouldn’t be used to defi ne design created “by

the computer” – it should be used to describe “through the computer”

instead. In a broad sense, any design that is visualized through a computer

can be considered generative, but the usage of the term becomes more

appropriate when describing automated processes - the computer is

programmed to build automatically certain visuals according to a set of

variables and pre defi ned laws with no need of human interference.

��

��

��

��

��

��

50

“…the language of the computer is the language of technology, not the language of design. Without a knowledge of design, the computer (like the pencil) is more than useless, for it is capable of producing enough superfl uous material to create the illusion, that one is inventing when in fact, one is merely producing varia-tions of a theme, often of nothingness.”

— Paul Rand, Design Form and Chaos, 1993

53

Praxis

The concepts of computational design need to be applied somehow; In

order to transform abstract concepts into real projects, tools need to be

used.

The computer is obviously the most necessary tool for the task. It needs

to feature a screen – a device capable of displaying an array of pixels,

light elements that are lit or not to build shapes and colors. It also needs

to feature input devices, like keyboard, mouse or joystick. Finally, it

needs the software, a logical layer between the computer circuit and

the designer. The choice of software is a controversial issue: different

packages offer completely different experiences for the designer and

for the audience. Also, every software title imprints different aesthetical

details to the pieces generated by them.

A software title that helps the creation of visual experiences will act as an

interpreter, a translator between the designer and the machine. Different

interpreters will build the translated content in different ways; the overall

meaning might be the same, but the lexical construction will differ. Also,

software packages use libraries of functions – pre written formulas that

help the development of a visual piece.

When the functionality of pre-written libraries grow, an effect called

pasteurization might happen: little individual differences in the code

between one author to another are suppressed; the code, thus the visual

(since one generates the other) gain a very similar aesthetic aspect. They

look the same. Commercial packages often break the barrier of canned

aesthetic subtleties and offer completely industrialized, homogenous

visual solutions - templates. Less, easier work – but evident less visual

value in design.

54

Forms of Code

Canned or artisan, the software tools provide a clean slate for creativity.

There are many options to design through algorithm and these options

have mainly three categories: compiled, scripted (aka interpreted) or

markup languages.

Their defi nition is a little fuzzy; the main idea is that a code written in

a compiled language is converted to machine language – “compiled” -

before it is distributed and executed. The fi nal product does not contain

the original written code, but a translated machine code. For example,

C and Pascal are purely compiled languages. C++ and Java are semi-

compiled, they stand in an intermediate structure between compiling and

scripting modes.

Scripting languages are interpreted at run time; the code is converted

to machine instructions at the same time the program is running. The

distributed/executed code contains the original language written by the

programmer, and it needs the support of a compiled structure to receive.

The performance of scripting languages is worse than purely compiled

ones, but its fl exibility is greater; Postscript, JavaScript and Flash’s

ActionScript are popular examples of scripting languages.

Markup language is the name for content that is tagged to be correctly

displayed and interpreted. HTML and XML are markup languages – texts,

pictures or any data with the correct tags for the future interpretation.

Forms of Image

Another important distinction to be noted in graphic software is the defi -

nition of bitmap and vector images.

Vector graphics are shapes defi ned in the computer memory by the

formulas of its form, the formulas of how lines and curves are posi-

tioned and connected. They became very popular on the web thanks to

Macromedia Flash, and very popular in print design thanks to Adobe

Postscript. Vectors can be written as a map of coordinates in two or

three dimensions, creating 3d models. The vector formulas need to be

55

converted to pixels before being shown on the screen; depending on

the software that is interpreting them, different rendering results are

noted, different visual styles are achieved. Pixels are square elements,

and they’re required to represent points in curved shapes; The rendering

engine needs to decide which pixels should fi t in the curved lines, and also

decide how the jagged edges can be smoothed (also know as antialiasing)

to complement the illusion of curviness

Raster (or bitmapped) graphics are visual fi les where an array of pixel color

values is stored; all the pixel values together form an image. Photographs

are examples of bitmapped images; bitmaps can be compressed through

algorithms that perform pixel statistic operations, discarding information

that it judges not relevant to the image and making them smaller. Bitmaps

are already made out of pixel information, so unlike vectors they don’t

need to be converted to be displayed on screen.

Software

The most common software tools available today for the different code

form categories are:

• Adobe Photoshop/Illustrator/After Effects

the Adobe software titles are used as graphic production tools, not

coding environments; but the reality is that algorithms can be written

a vector image, and a blown-upof its pixel representation.

56

and implemented on then. After Effects, the broadcast motion graphics

standard has support to the common JavaScript. Illustrator can

import postscript fi les that can be generated through code; Photoshop

also can have plug-ins developed, automate functions through scripts

or deal with postscript fi les. All these coding doors can be taken as

advantage in the creative framework.

• Macromedia Director

One of the oldest multimedia authoring platforms; Director is fi rst of

all, an aggregator: it can unite almost any kind of media asset inside a

project. Director has a powerful, but very extravagant code language

called Lingo - extravagant because its syntax is very different from

the coding standards used by other popular multimedia applications.

Director has full support for real time, hardware accelerated 3D

graphics and is largely used in complex networked projects, where

fl ash is not powerful enough.

• Macromedia Flash

Flash is one of the most successful pieces of software in history; the

initial plug-in was designed to visualize vector graphics in a web

browser, and animate them through tiny fi le sizes. The original

l996 software called “CelAnimator” evolved, having more and more

features added every year. Flash is a program that can be used in in-

numerous ways – as a linear animator, as an illustrator or as a creator

of applications and web sites. Flash has a scripting language called

ActionScript, in part responsible by the actual movement of designers

working through code. It is necessary to learn ActionScript in order

to build interactive movies; the basic scripting of a site navigation

is an easy task for non programmers – which mean a smooth tran-

sition from traditional design to the possibilities of design through

code. ActionScript can be very simple, but it is also capable of doing

complex tasks as the user learns more coding techniques. There are

limits in fl ash though; the package still does not offer control of the

actual pixels being drawn on screen; the user can only code through

vectors, and the closed fl ash engine process the conversion to screen

pixels.

57

• Java

Java was created in 1995; its initial concept was to be an extremely

portable language that could be executed in any computational device

– from desktop computers to kitchen fridges. The Java-enabled

consumer electronics were never fully employed, with the exception

of cell phones, but the technology turned out to be very useful for

network applications – including the web. Java is a programming

language modeled after the Object Oriented concept, and is con-

sidered a “semi compiled” language, designed to be easily ported to

different computer systems.

• Processing

Processing a programming language created aiming artistic and

visual applications developed through code. Programs generated

through Processing are usually called sketches, and can be run in any

computer platform, or over the internet. Processing is built over Java,

having added its own graphic library and functions. Java language

has a steep learning curve, and requires lots of technical background.

Processing does not have a graphical input interface; the sketch is

designed in a textual way, but the logic within it is very visual, making

sense for visual oriented users, as artists and designers. Processing

grants pixel-level control; the screen array can be manipulated

directly. Also, it has great extensibility through additional libraries

– like sound, midi, HTML or physical interfaces. Almost anything can

be integrated into a artistic sketch project.

Every software package demands time from its user to fully learn all the

concepts, functions and possibilities. The technology of the future is almost

unpredictable, making hard to decide which platform will have more ac-

ceptance and more capabilities in few years. Macromedia Director, for

example, was really popular for the CD-ROM production in the nineties;

its usage has been decreasing, being now restricted to some specifi c 3D

projects – and it’s likely to become less popular in few years. Other factor

to consider when learning a language is the “commercial versus free”

software factor; commercial packages usually have, at least in theory,

58

more technical support, because they’re products being sold to customers;

the designer is paying the developers. In fact, free software communities

end up having a good network of users who support each other, even in

a better way that paid application service. The real advantage on com-

mercial titles is the marketing that convinces users to adopt means that

will allow projects to be seen. Recently, Microsoft decided to cut the free

Java as a pre-installed plug-in for the Explorer browser; meanwhile, the

Macromedia Flash player still comes pre-installed, and is present in 95%

of the user browsers.

Using Java/Processing as a creative tool also features the advantage of

controlling the visual rendering process to a low level depth. As said

before, pre-programmed functions end up having an aesthetic infl uence

to the visual product; programs like Processing allow the designer

to customize the aesthetic imprint by writing their own set of visual

rendering functions.

still from Bézier Ballet, by Mario Klingemann (http://www.quasimondo.com).

Unhappy with the Processing default antialiasing algorithm, Klingerman wrote his own, capable of displaying the smoothness

and delicacy of thin bézier lines.

59

Mathematical Tools

Each software package has its own features and its own coding syntax to

be learned. But one thing is common among them all: Math.

It is necessary to know some math concepts in order to create visual

experiences through algorithm; and this is one of the reasons that people

are driven away from this kind of creative process. Art education has been

far from Math, and mathematical ideas are seen as impossible problems

by most of designers.

Happily, the truth is that the necessary math for visual creations is very

tangible, and all it requires is the high school math remembrance. 95%

of the creative visual work produced through algorithm today employs

high school level math only, including algebra, trigonometry, physics and

geometry.

The key is not to know complex math; the key is to use creatively the math

you already know. Math is a tool, and the complexity of the tool never

meant the quality of the creation – neither in arts or sciences.

The Leica camera used by Cartier Bresson had only basic photographic functions. It is a quality simple instrument, the tool of Bresson’s beautiful photographic expression.

60

Remembering high school

What is the basic math a designer is supposed to know?

A good general review at the high school books is recommended;

sometimes, easy formulas or concepts that may be forgotten can help

to transform an idea into a visual. There are some topics that deserve

special attention; a good foundation on trigonometry, for example, will

solve a great number of issues related to circular motion and visual oscil-

lation. The Pythagorean Theorem, that solves the right triangle lengths

(c2 = a2+b2) , will also solve questions regarding positioning, distance and

linear motion along the space of a visual piece. Basic operators (addition,

subtraction, division, multiplication) can also have creative potentials

- since a block of code will loop indefi nitely, and a number can go through

an operation and change its value on each cycle. The modulo (the operator

that returns the remainder of the division between two numbers) is one

of the forgotten high school arithmetic topics that can have a great use on

computation – modulo values are cyclic, they repeat themselves when the

operand is constantly incremented in a loop. Colors, forms and position

can be cycled along the modulo result. The author of this document knows

very little math beyond the topics related above – and even so, produced a

series of intriguing mathematical visuals; the best advice to overcome the

Math-phobia is to roll up the sleeves and work on visual projects – learn

the math that will solve each specifi c visual inquiry, one at a time.

Following, a code example built in Processing that shows the application

of modulo and trigonometry; a box rotates in circular motion – achieved

by calculating the x and y coordinates according to trigonometric rules;

its color switches to black and white according to the modulo value of the

loop cycle. The sketch can be found on the attached DVD under the name

Trigonometry.pde.

“Art and Mathematics are complementary different; that is to say: they differ each other just as much as they complement each other.”

— Karl Gerstner

61

The Code Language

There’s not a universal language for computer coding; each coding

platform has its own syntax. Different languages can share similarities,

but they have different ways to handle the code structure, making them

unique - and requiring unique training.

Someone who’s aspiring to write code should choose which language fi ts

best to his needs; for commercial internet creation, Flash is surely the

best choice today. For artistic experimentation, Java and C++ are better

options, since they offer less limitations and more performance than

Flash. The Processing language has a lot of advantages for the artistic

fi eld, including strong community support and a balanced learning curve.

There are many other options – Python, C#, Objective C… all computer

languages can be a source for creative work.

There’s also no single method of learning the code syntax; each author

proposes a different path through the functions and commands of each

platform. A recommendation for the learning process of a visual-oriented

user can be something like:

fl oat counter;

int radius;

fl oat x, y;

void setup() {

framerate(20);

size(200,200);

counter = 0;

radius = 50;

}

void draw() {

background(122);

if(counter%2==0) { fi ll(0); }

else { fi ll(255); }

x = sin(counter/10)*radius;

y = cos(counter/10)*radius;

rect (90+ x, 90+ y, 20, 20);

counter = counter+1;

}

create a loop counter variable:

create a circle radius variable:

create the x and y coordinates variables:

confi gure the piece:

set the velocity of the animation:

set the dimensions of the sketch:

set the initial value of the counter:

set the value of the circle radius:

end of the confi guration block:

loop starting point:

set background color:

modulo operation:

if the value is even, fi ll the square with black:

if odd, fi ll it white:

x axis value according to trigonometry circle:

y axis value according to trigonometry circle:

set the rectangle coordinates and size;

increase the counter value:

end of the loop block:

62

• learn to compose a basic code structure – setup and loop blocks.

• create and confi gure the visual space for a piece (the stage)

• learn the methods of drawing basic shapes;

• learn the methods of tweaking the attributes – color, size, rotation

– of those shapes;

• learn how to move things around.

• learn how to make the user interact with elements.

This sequence would start out of graphic functions, instead, for example,

the common foundation on Object Oriented Programming – that involves

more abstract and less visual thinking.

The Processing platform has a lot of support for learning from the artistic

point of view; the language reference includes simple graphic examples

on the syntax of each topic at http://www.processing.org/learning and

http://www.processing.org/reference.

63

The Praxis of Coding

The process of writing code is similar to the process of writing a narrative

text.

A narration tells a story; there are characters, each character has a role

and things happen as the story develops, coming to an end. The same

narrative story can be written in different styles, different languages;

it can be written in a very erudite form, or it can have a teen literature

lexicon.

A computer code with a determined objective also can have different

styles, and different structures. The style and structure of coding can

be very personal; the way variables and objects are named, and how an

algorithm is written or a problem is solved. But there are principles that

should be followed.

The basic principles of coding are simplicity, clarity, generality and au-

tomation.

By defi nition, computation is a problem solving task. And there is usually

more than one way to solve a problem; the principle of simplicity is

that the simpler way to solve a problem is the best – less computational

resources are allocated, leaving more room for other operations.

Clarity is an issue for both man and machine; code should be clear for

other people rather than the author to read it, and for the machine to un-

derstand it. The clarity is achieved through good naming practices, using

parentheses to clear ambiguities (sometimes its use is not mandatory,

but helps the human reading), commenting the chunks of code, and

describing any relevant information in the program body. Another item

for clarity is the avoidance of contradictions; for example, the usage of

lower and uppercase. A variable named “Test” and a variable named

“test” are completely different in a case sensitive environment. If a piece

of code is using a variable in lowercase, all references to it should also be

in lowercase.

Generality is a principle that follows the spirit of object oriented pro-

gramming. A program can have very specifi c functions for each task;

64

sometimes, these functions are similar, and could have been written as

one and reused. General functions allocate less memory in a computer

system and allows better performance of the code.

Automation is a principle where the computer should handle repetitive

tasks, not the designer. It is an idea related to the principle of generality;

Instead of writing a commands or functions that perform a similar task

hundreds of times, it is better to write one function that operate the same

way with different parameters, saving time on the development and

making the debugging process easier.

A good practice for someone who is not comfortable with the task of

coding (or for someone with a very complex task in hands) is to fi rst write

the program statements in plain English; list all the actions, variables

and describe all possible functions. With a cohesive and well structured

English text in hands, just translate it to the programming language

syntax – and the code is done.

67

Experience

“Graphic design which evokes the symmetria of Vitruvius, the dynamic symmetry of Hambidge, the asymmetry of Mondrian; which is a good gestalt, generated by intuition or by computer, by invention or by a system of coordinates is not good design if it does not communicate”

— Paul Rand

A Visual Experience involves the audience; designing a visual experience

does not contain only the “what to do” issue and the “how to do” issue; it

also contains the “whom to do”, the concern on how a visual piece will be

experienced by the audience.

The experience is an individual process; each person responds to a com-

munication stimulus in a particular form. The creation of visual expe-

riences deal with probabilities; defi ning the most probable response to

each presented stimulus, and planning how the visual piece will develop

according to the most probable situations.

There’s no real passive audience; if a visual piece is not interactive, there’s

always a choice for someone to see it or not; to ignore it or enjoy it. The

act of fl ip a page or not, close a browser window or not is a basic level of

interactivity.

Interaction is any reaction to the design stimulus.

A positive reaction is the successful communication. The designer should

compose a piece in visual terms but also in interactive terms; how a piece

will be originally seen? Is there a waiting time to see it? How much infor-

mation is given at each step, each section, each second?

Dosage: the concentration of information on every design node, part,

section, screen – considering information every visual, aural or any media

output. The design of interaction deals with the dosage of how and when

things will be experienced along a piece.

68

Regarding the visual terms, the shape of information should raise identity.

Identity means that the each person in the audience will recognize aes-

thetical ideas in common between him and the presented experience. The

recognition can be simple as a “I like it” response – and it does not mean

that the user identifi es the output as something already seen before,

but as something in his personal taste, a match to his preferences. The

aesthetic values should be shared between both ends of communication,

the designer and the audience.

The development of visual experiences through algorithm opens doors

of limitless customization, interactivity and the generation of customized

information channels: each piece can be designed and react according

to the viewer. Despite of the unpredictability of a human response, the

audience should be acknowledged during the creation process - the

designer should not lock himself in an ivory tower, design is about public

experience – design is about communication.

69

70

71

Case Studies

The following projects were developed during the MFA program at

MassArt; they relate, identify and explore issues on the task of articu-

lating visual experiences, and they are the product of a designer – whose

education was centered in humanities – dealing with computation as a

tool for the visual domain.

72

73

Sound in Space

“Music is geometry in time”

— Arthur Honegger (1892-1955, Swiss composer)

What’s the relation between the aural and visual domains?

Is there an image for sound?

Sound is powerful enough to survive without our eyes. Music is a legitimate

expressway from and to our souls; music strikes the heart with no need

of visual alliances. At the same time, once we add a visual component to

sound, a relationship is formed - visual and aural domains stick together,

building an undeniable bond.

The bond can be formed in two ways; the fi rst is by converting the

character of the musical signal into a visual representation; Apple Itunes

does this very well. Its visualizations uses the pitch and volume to feed

graphical algorithms. The second way evokes the meaning of music along

with the meanings of the visual domain; this idea is seen (and heard) in

cinema. A dramatic scene has a dramatic soundtrack; “dramatic” is an

adjective for the musical meaning; the pitches and volumes alone are not

dramatic – but the musical movement altogether can be.

The relationship of visual and aural has been explored throughout history

in many different ways; in eastern civilization, one of the oldest scientifi c

approaches to it comes from Isaac Newton. In 1672, he studied the color

spectrum and related 7 colors to 7 musical notes. Later, Louis Bertrand

Castel built the fi rst “visual keyboard” – the Clavecin Oculaire – that

displayed a different color to each played pitch. The idea of a keyboard

that would display colors for each note reappeared in different places and

moments in time; there was, though, never a fi rm proof that a certain

74

color is a certain pitch. The relationship always had an arbitrary element

in its formation. Different color/pitch tables were developed, and many

authors agree on the color distributions – but there was never a scientifi c

standard. Intuition and feeling, personal factors, remain present.

Besides color, sound always had a strong relation to math and geometry.

The notable study of this relationship is harmony. Harmony is the study

of the simultaneity of sounds; Pythagoras, more than 2,500 years ago

started to study why some sounds had a pleasant character when played

together. Pythagoras studies had a strong mathematical sense, and he

established the concept of intervals.

Each musical note is a vibration; each vibration has a frequency. Our

perception of pitches, or musical notes, has a cyclic nature – every time

a frequency doubles, we perceive it as the same pitch, on a higher tone.

It’s called an octave – the space between two frequencies with the same

note. Numerically, the octave interval is notated as 2:1 proportion– the

frequency of the interval note is 2 times bigger than the whole funda-

mental note. A fundamental note can be divided in any other proportion,

and some of them are used as standards in music, like 3:2 (1.5 times bigger

frequency than the fundamental - fi fth interval), 4:3 (fourth interval), 5:4

(major third), 6:5 (minor third), so on and so forth.

the proportional lenght between some musical intervals.

��

��

��

��

��

��

75

A mathematician called Jules Lissajous developed an experiment in 1857

that projected a vibration movement through a light dot in a screen. He

managed to refl ect the light beam through 2 mirrors, each one vibrating

in an axis, vertical and horizontal. The resulting projected shapes became

known as Lissajous Figures

Different vibrations could be applied to the mirrors, producing different

shapes; musical notes are vibrations, so different pitches could be

reproduced and visualized - and when different pitches were projected

together, musical intervals could be seen.

The Lissajous fi gure might be the most scientifi c approach to a visual

representation of sound; it is based on the physical properties of musical

notes that cannot be easily contested like the previous color/sound ex-

periments.

original Lissajous experiment

76

The Chordinator

The Chordinator re-creates and expands Lissajous experiment as an

interactive computer application. Lissajous projection was limited to one

single interval; two notes, because his physical projection system could

only be bi-dimensional. The starting point of the Chordinator is to take

advantage that a computer system can create the illusion of depth easily.

There are 3 spatial dimensions that the human perception can discern:

height, width and depth. Musical chords are 3 (or more) notes played

together; two intervals in the same period of time, in a chord structure (

root, II degree, III degree, the button labels) The idea is to assign a spatial

dimension to a sound frequency, and explore the plasticity of sound as a

sculpture.

Chordinator interface screen

77

The sketch allows the independent control of the intensity of each chord

component, as well as its phase. Phase can be understood as the coordi-

nation point between sound waves

To help the sense of 3 dimensional space, a cube was added marking the

capping of the volume; rotation was added so the point of view can change

over time, allowing a full visualization of sound in 3D.

Despite of the strong sound concept, the Chordinator experience aims to

sculpt a visual structure over time: the keyboard is controlling a form in

space, not music.

Production notes

The Chordinator employs a very simple programming concept: parametric

equations, where independent equations drive different coordinate axis.

The musical note visual component is a graph plotted through time; each

coordinate is being fed by an algorithm that determines the amplitude,

pitch and phase of sound, independently. There’s an independent

module that creates the cube and another that controls the interface. The

algorithm follows trigonometric principles, where the sound vibration can

be expressed as a sine wave, with defi ned amplitude, length and period.

The 3D visualization was built in Processing, and the keyboard interface

in Flash. they both work together in a browser window, allowing this

visual experience to be shared online.

��

phase example: three sound waves with same amplitude and wavelenght with different alignments in time.

78

The Harmonographer

The harmonograph was an invention built and sold in the nineteen

century and it consisted in pendulums attached to a pen; the pendulum

movement set the pen coordinates, drawing the resultant of two (or

three) harmonic motions. The fi nal drawing is a frequency interval, like

the Lissajous experiment; the pendulums also had the nice feature of

simulate the damping of a musical note, the slow decay to silence, thanks

to the mechanic energy that is dissipated within the motion. Another

feature of the harmonograph was the use of rotary motion; it added a

new dimension and perspective to the visualization. The frequencies were

more dimensional, more representative of their physical nature in space

han the Lissajous projection.

Harmonographer interface screen

79

There was a strong relation between musical harmony and visual

harmony in the interval drawings of the Harmonograph; perfect intervals

were symmetrical and visually stable; numerical approximations in the

frequencies produced more unbalanced visuals, with a more intricate

nature.

The Harmonographer project is the translation of the physical device to

the computer; it has the feature of dialing in the musical notes, obtaining

the correct intervals. It also simulates the musical damping, and the

phase oscillation.

The images produced by the Harmonographer should be understood as

drawn from the outer edges (maximum volume) to the center (silence)

- the path of decay. The constant rotation of the drawing has a purpose:

show the interval at all possible sound coordination points, and its

circular motion comes from the phase value. The phase is a cyclic event (it

repeats according to the length of a wave), so there’s a constant rotation

impression to the coordinates.

The sketch also features acceleration from one interval to another

and variable interval change speed; the function of both is to induce a

smooth transition between different visuals, letting the user know what

is happening from one note to another – or just fl y through all possi-

bilities.

an original Harmonograph

80

A “Portamento” button is available to turn on the infi nite sliding between

notes; in the portamento mode, the interval grows eternally, and the user

can keep on watching and observing how the intervals grow and their

visual elements change with the growth.

The main objective of the Harmonographer is let the user explore the

interval relations through their visual representation; or simply explore

the visual patterns and their mandala-like formations, tweaking the

possible graphical formations that the algorithm can build.

A special version of the Harmonographer was also built, that allows the

user to export the drawing as a vector fi le, that can be used in any graphic

application or printed. The user can also save the drawings as bitmaps,

and create video animations.

Production Notes

The coding behind the harmonographer is based on parametric equations,

independent plot instructions for x and y values. The code is slightly

more complicated than the Chordinator, since there are formulas for the

damping - that use the harmonic damping formula from physics books.

The damping was implemented as an algorithm, and the phase rotary

motion was added as well - using again the trigonometric circle concept.

The code has separate sets of acceleration and portamento functions,

and all the interface commands function, along with the Flash virtual-

keyboard element.

81

The Near Dorian Experience

How can a musical piece be visualized through the harmonograph?

To answer this question, a video called The Near Dorian Experience was

developed, visualizing an original music composition.

The musical soundtrack intentions for the Near Dorian were to work

with few and strong musical intervals, and create a hypnotic trance of

repetition - but at the same time calm and meditative. There’s one guitar

with a mellow, woody tone and another with a twang slide sound. The

percussion is nothing more than a triangle being hit twice in the music

and the base sound is one single chord being played in a Hammond

organ, oscillating through rotating speakers. The intervals explored in

the song were pre selected through their visuals – the interesting visuals

had a granted place. The video has layers for the different instruments

and their visuals; color was added, in the form of blue tones denoting

the cold electric resonance, and yellow/orange denoting the warmness

of the guitar strings. The white denotes the purity of the bell sound, the

Near Dorian Experience frame still

82

moment of illumination. Of course, this description of the soundtrack is

from a author standpoint, and the actual experience of musical meanings

may vary.

The piece shows the coordination between aural and visual intervals. The

title, Dorian, is a musical mode of the natural musical scale, known by its

simplicity. Since the song escapes the scale in a part, and the visual escapes

the simplicity due to the overlaying, it’s a near Dorian experience.

This project has the conceptual importance of being a completely

authorial project in new media – the author developed the expression

in all domains: visual, temporal, musical, and logical. All the pieces were

original and neither contained parts from other sources or projects.

Beyond that, it has the proof of possibility to develop a concept in the all

domains at the same time.

A software application was created to achieve the visual of the video,

and it couldn’t be done in another way; even if a physical harmonograph

was constructed, for example, it wouldn’t animate the phase variations

of rotating speakers - there are features that can only be achieved in a

logical medium.

The sound in space project was essentially pseudo-scientifi c; it had sci-

entifi c fundaments, but embraced creative freedom to show a personal

view of the spatial information of music. The personal view was expressed

through a computer program that helped the development of a song and a

video - the freedom of articulation came through algorithms.

83

85

Form Systems

A visual composition is formed by individual elements; these elements

interact. The study of how and why the elements in a composition interact

is the subject of visual language.

Many artists and writers published their research on the nature and prop-

erties of visual language, and one of these authors is Karl Gerstner, Swiss

painter and graphic designer. Gerstner has a remarkable point of view

regarding form and color – according to him, they’re body and spirit.

Gerstner considered math “an aid to ultimate master of one’s medium”.

His most notable characteristic is the usage of structure as a mean of

expression; many of Gerstner art pieces were not fi xed compositions,

they were fi xed formulas where the components could be interchanged.

Although Gerstner didn’t employed computers, he created artistic algo-

rithms; his pictures are variation machines. This computational spirit is

the inspiration for the series of form systems experiments in which the

nature of variation is explored.

Gerstner work is the inspiration for a series of experiments, where

concepts of the computational medium that connect with the idea of form

as a system are explored.

The Monkey+Gerstner and No_More_Mondrian

A good example of Gerstner’s art is the Carro 64 piece. It consists in

64 aluminum blocks, in 16 colors (a 16 step color gradient) that form a

“do-it- yourself” composition: the blocks can be arranged in any way. The

arrangements can follow symmetrical operations or be random; there are

trillions of possible combinations in the random mode, but the number

of options is smaller when compositing laws are inserted, forming deter-

86

mined color patterns.

As an experiment, I rebuilt Carro 64 as a java sketch – where the compos-

iting possibilities would be alternated but never repeated. When trans-

lated to the electronic medium, the blocks location exchange is effortless.

The canvas becomes an almost infi nite art generator, rebuilding the color

blocks in a different confi guration on each determined time interval.

Are the color compositions interesting? Eventually. Being random, there’s

a random possibility that an interesting or visually attractive composition

Carro 64 advertisement

87

will emerge. An interesting experiment was built on the web by the

software company AardAsNails, called Monkey Shakespeare Simulator1.

Launched in 2003, it proposes to prove the mathematical infi nite monkey

theorem, by the use of bulk grid computing and random letter string gen-

eration – a monkey simulator, that would run on each user computer.

“If you have enough monkeys banging randomly on typewriters, they will eventually type the works of William Shakespeare.”

The record so far is 24 letters from King Henry II – and it took only

2,737,850 million billion billion billion monkey-years to be completed.

“Eventually” is a tricky word that tends to the impossible. Maybe that’s

why Gerstner, even though creating an infi nite art form, exposed confi gu-

rations with carefully planned symmetric harmonies that required plenty

of artistic sensibility to be achieved.

The point of this experiment is to explore the role of authorship in the

so called generative design; the absence of an author’s guidance creates

a monkey situation for the machine. There’s value on the concept of an

eternal change art canvas, but the isolated result of the variation is not

guaranteed. In a form system, the author can create rules, directions

and behaviors that will channel the creation interesting levels, instead of

abandoning the visual dimension to its own random generative luck.

the original Carro 64, and one of its random computer recreations

1 http://user.tninet.se/~ecf599g/aardasnails/java/Monkey/webpages/index.html#results

An experiment with real monkeys was also conducted with hilarious results at

http://www.vivaria.net/experiments/notes/publication/NOTES_EN.pdf

abandoning the visual dimension to its own random generative luck.

88

The other side of a generative form system occurs when a complete, vast

set of rules is laid to control the unpredictable randomness. The uncertain

composition is tamed, but the absence of the monkey factor can also

create banal pieces that always look the same.

The predictability usually occurs on systems designed to mimic an

abstract process. During the early days of graphic computing, Michael

Noll developed experiments where a computer was programmed to

mimic Mondrian’s Composition with lines. Noll’s conclusion? Viewers of

his experiment could not tell whether the painting was created by the

human Mondrian or the electronic Mondrian.

Recreating Noll’s experiment in Java has brought me some interesting

conclusions; fi rst, it’s complex to mimic a human artistic behavior.

Second, it’s really easy to create a sketch that draws geometric forms in

random patterns, similar to many abstract paintings – but no abstract

painting is like that. Even simple geometric paintings deal with a complex

mental process of decision making: should a line be on the left or right?

Should a square be big, small or medium? What will be the horizontal to

vertical lines proportion?

To answer these questions, rules need to be written in the code. Rules that

determine average size, average proportion, placement limits, so on, so

forth. Even a minimalist painting mimic needs a great amount of rules;

the more rules, the closer the generated image will be to the original – but

it will also be more predictable. The image won’t bring new, interesting

formations that could exist outside the boundaries of a certain rule.

Any painting can have an infi nite set of rules – they can grow in com-

plexity, more the rules, less the randomness, more faithful to the original

generative design processor

89

artwork. For the No_More_Mondrian sketch only a limited set of rules

were written, a number enough to resemble the original experiment, and

the sketch runs forever always trying to add a new layer of “ruled” shapes.

Dull, as expected.

The Squircle

The key to creating generating form systems is striking a balance between

rigid rules and anarchic randomness.

The Squircle is a small experiment toward fi nding this balance. Based on

Gerstner’s Color-Form Continuum, in which he explored transitory se-

quential states between different shapes and colors, the Squircle explores

all the shapes that lie between a circle and a square and the visual compo-

sitions that derive from them.

The Squircle was built using the formula of Bézier curves - a common

tool for Postscript, CAD and vector illustration applications. Bézier curve

is a parametric cubic equation that plots a path with initial and fi nal

points infl uenced by control points along the way. The control points can

be tweaked to change the overall shape of the curve. By coding a closed

shape with symmetrical control point changes, the result is a regular

polygon whose sides bend and morph in all possible positions - the

clash of generations: the original Noll experiment and the actual Java recreation.

90

screen captures from The Squircle

and Gerstner’sColor Form

structure model

91

square and the circle are the two extremes of the possible forms. Instead

of copying a Gerstner piece or simply creating all possible shapes in a

random fashion, the Squircle uses an oscillatory pulse that changes the

control points position, and the amplitude of the change can be set by the

mouse movement. The sole purpose of the piece is for the user to enjoy

creating forms with symmetrical aspect and experiencing the possibilities

between two basic geometric shapes.

Interactivity is a useful tool when there’s a quest to create an endless form

system that is not random; user interactivity determines a composition in

the Squircle, that is a unique and endless, but not random.

The Anaconda Paint

In another experiment, Anaconda Paint, the user builds a volumed form

according to mouse coordinates. The visual composition is obtained by

varying the color and position of a circle, with either a dented or plain

stroke texture. The illusion of a dimensional painting is created in a fl at

surface by overlapping circles through time. To enhance the organic

aspect of the composition, the shape moves with acceleration toward the

cursor, avoiding abrupt movements.

It looks better in motion.

92

The “trick” that makes the Anaconda Paint interesting is the color shifting;

all the colors in the screen are constantly changing, in a cyclic way. Each

circle is assigned with a different initial color – and when all the screen

colors cycle, the illusion of volume is reinforced.

The Anaconda visual experience is supposed to be fun to play with, letting

the user create endless snake fi gures. This piece is completely different

from the Monkey+Gerstner, but both have the same concept: endless

compositions, based on a fi xed shape, with multiple instances and varied

color and location. There are innumerous ways to translate a single

concept.

Roto-Champ

The idea of keeping a shape fi xed, but to vary its color, size, and position

over time is full of possibilities; the Roto-Champ experiment explored

another point of view on this theme. The project is based on Marcel

Duchamp Roto-reliefs, a series of gramophone discs painted with patterns

that would rotate and create unexpected kinetic results. The Roto-Champ

is an exploration of Duchamp’s concept translated into the computational

medium.

The shapes have fi xed form and color, and rotate at a constant speed.

Illusions of volume and form distortion are created by the visual inter-

action of the concentric rings, just like the original piece.

The Outline Anaconda

93

The experiment has two different approaches; the fi rst regards the

geometric nature of the circle. The circle is a pure mathematical form,

and its presence in a plain color background has a strong abstract and

ethereal sense. I questioned how to add more physical, concrete value

to this form - and the answer was to tweak the relative size and rotation

of the rings. The values inserted were transformed according to a value

very common to Botanic: golden section . The sacred mathematical pro-

portion is found on the growth of plants, on the patterns of seeds and on

nearly every natural organic form. Both size and rotation would respond

to organic values, linking the ethereal form to a more physical dynamic.

Marcel Duchamp invites you for the Roto-relief experienceRoto-relief experienceRoto-relief

Roto-Champ: motion blur version

“the proportion resulting from the division of a straight line into two parts so that the ratio of the whole to

the larger part is the same as the ratio of the larger part to the smaller” – Oxford English Dictionary

94

The second approach deals with interactivity; with one mouse coordinate,

the user can change the perspective of the piece – something that would

happen in the original Duchamp piece, since the perspective would change

according to the position of the viewer. By visually cropping parts of the

concentric fi gure, the illusion is changed and different visual impressions

occur. The other mouse coordinate was used to displace the center of each

ring in relation of its inner sibling, generating then different frequency

patterns, and different aesthetic experiences.

Roto-Champ: golden proportion and standard versions in perspective.

95

My Friend Jong

Form systems can be taken to extremes; as in the case of “strange

attractor” visuals. This kind of visual has been explored by a number of

designers and scientists, and it is known by algorithms that plot points

with an apparent random, chaotic, order but that “gravitate” around

certain fi xed values (hence the name attractor), generating interesting

visuals. My own recreation of a strange attractor used the Peter de Jong

algorithm, written in 1991 by Paul Bourke. It was chosen because of its

simplicity - a small mathematical abstraction that can lead to complex,

visually rich results. The Jong algorithm simply subtracts a cosine from a

sine value, in both screen axis (x and y).

The trick is: given a starting point, the sine and cosine values are calcu-

lated based on the previous operation result, in a loop. This loop is called

iteration, and the number of iterations can be predefi ned – ranging from

1 pair of coordinates to billions. Organic patterns emerge from the plotted

dots– the cold mathematical abstraction is transformed in a shape that

looks and behaves like organic dynamic forces, as the smoke of an incense

stick. Given a smaller (few thousand) iteration cycles, the attractor can

be animated in real time, or respond to user interaction, like mouse

movement. Attractors are systems with little visual control (they have an

unpredictable nature) but are a good playground where small changes can

be multiplied and re applied millions of times. The concept of iteration

can be used as a creative tool – there’s not much control over the form

itself, but on how the form will emerge and unfold through iteration. This

is an important abstraction on the realm of generating images through

code: sometimes, a visual can’t be grasped; it has to be tamed through the

numbers that feed its creation.

Peter de Jong - algorithm by Paul Bourke.

xn+1

= sin(a*yn) - cos(b*x

n)

yn+1

= sin(c*xn) - cos(d*y

n)

96

My friend Jong: 150,000,000,000 iterations. (I counted them myself to double check if there was any missing.)

99

Riley Patterns

Op art is an artistic movement that blossomed in the middle sixties,

through the work of artists like Victor Vasarely and Bridget Riley. The

main characteristic of op art is a sensorial overload; the perception is

challenged with tricky impressions of volume, color and motion. Op art is

extremely kinetic - but in a perceptual, and not physical level.

Another trademark of op art is the creative use of patterns; visual

rhythms with distortions and modulations that conduct the composition

experience. The use of patterns and modulations usually follow a defi ned

set of rules – a strong sense of algorithm.

Riley and Vasarely works were not computational – their creations were

set in a time when computers where scientifi c secrets. My intention is

to investigate how optical visual patterns can be created and modulated

through code, and explore the computer’s advantage as a variation

machine: visual ideas can be developed in endless ways, especially when

the idea has an algorithmic nature. What are the changes in op art when

it is reclaimed by the computational medium?

Baba O’ Riley

“The art of tomorrow will be a common collective treasure or it will not be art at all.”

-Victor Vasarely, 1960

Vasarely professed the mass media possibilities at an early date. In which

ways can art be collective?

Baba O’ Riley is an attempt to answer this question, following Gerstner’s

spirit of having structure as art. Baba O’ Riley contains no composition: it

contains the structure for endless compositions. The authorship is shared:

100

the designer authors the aesthetic framework; the audience experiences

the creation of a composition from that framework. It’s a true form of

collective art – everyone can do it.

The user has the ability of drawing lines and circles, with different color

frequencies - always alternating black and white. Even being just a

computer application experience, the decision process on how and where

the drawing elements should be placed is similar to the physical painting

creative process. The medium is different, but the abstract thought

process is alike

The fl exibility on instantaneous composition is interesting for the artist/

designer; it allows painless experimentation. The successful tryouts

can be saved, the unsuccessful discarded – and the saved ones can even

become sketches for physical paintings. One of the major diffi culties of

creating physical “real world” op art creations is the technical production

of visual patterns; they require a lot of effort and planning. A computer

application can help a lot the process of planning, easing the technical

barriers and amplifying the creative freedom.

Baba o’Riley interface screen

101

Development

Baba O’ Riley was developed from scratch, programmed into the

Processing environment. It involved an interesting aspect of computer

graphics: drawing a circle. Processing was chosen because it permits

great control of the screen; pixels can be drawn and manipulated directly,

feature absent in programs like Macromedia Flash.

Designers take the computational circle drawing for granted; in Photoshop,

Illustrator, Flash or even on the simple Windows Paint, there’s a tool that

allows the cursor to draw the basic fi gure. But how does the tool achieve

that work?

That was the starting point of the investigation. Pixels are square; the

screen is a big group of squares. How can such a round fi gure exist in this

harsh environment?

The fi rst answer: it can’t. A completely perfect abstract curve cannot be

drawn to the square based screen. It’s necessary to build a function that

returns the coordinates of the pixels that better match the path of a curve.

Curve lines on a screen are approximations; thanks to the resolution, they

look acceptably good.

The principles of trigonometry grant the knowledge of discovering such

coordinates; once the radius of the circle is known, the Pythagorean

Theorem can be applied to every pixel in the x coordinate – it will

return a matching y, creating coordinate pairs – the fi rst step for a circle

algorithm.

Unfortunately, the resulting curve line is not continuous; the frequency of

y coordinate steps wears thin toward the edges – the pixels’ imprecision

appears. The second step of the algorithm is to fi ll those gaps – with

square pixels failing to follow a curved path

102

straight lines. The result is a perfect circle – built through inconstant

coordinates and straight lines, in a computational illusion. This is the

trick that Photoshop does all the time for its users.

The third aspect of the algorithm is to build the curve with the right color.

It fi rst evaluates the present color in the x/y coordinate, and redraws

it with the opposite value – the law of high contrast is followed all the

time.

The remainer of the code deals with mouse/keyboard response and

interface parameters. The program is very simple, and although there are

much better techniques of implementing the idea, it uses the simpler ones

in an extensively commented code. Only one basic mathematical concept

was employed – proofi ng that just a little math is already very useful to

express ideas through code.

The Baba O’Riley project was a success in terms of audience – the public

enjoyed the online experience (2,000 unique visitors so far) and it shows

how an interactive art project can be technically simple, but visually in-

teresting at the same time.

Current; Flow

One of the fi rst computer graphic images in history, along with Mondrian

experiments, was a recreation of Riley’s “Current” piece, built by A.

Michael Noll, called “Ninety Parallel Sinusoids with Linearly Increasing

Period”2. I decided to explore the Noll’s experiment, due to its historical

value and also due to its visual value; Current one of the most interesting

op art paintings.

Current consists in a repetition (for Riley, repetition is a mean for ampli-

fi cation) of waves with increasing periods (the length of each vibration),

and constant amplitude (the height of the wave). It’s a very well math-

ematically defi ned piece, and maybe that’s the reason with Noll chose it

for his early computer experiments.

The objective of the fi rst experiment was to explore the multiplicity of

means to an end; Noll’s experiment was rebuilt through different com-

2 Noll’s version was actually a recreation of Plate 3, another painting from the Current series.

103

original Noll experiment

line polygons Catmull-Rom curves

Bézier curves pixels

Current: many means to one form.

104

putational techniques – different processes for a single end. The visual

sketches were built using line polygons, Catmull-Rom splines, Bézier

Curves, and fi nally, direct pixel manipulation. Also, the mouse coordi-

nates were mapped to tweak amplitude and wavelength settings, helping

to mark visually appealing values.

Each technique has strong and weak features. The fi rst sketch used line

polygons, which are fast and easy to manipulate, but are not real curves

– they’re small line segments that resemble a curve. Working with

straight lines requires less processing power, but the fake nature of the

curves might become visible. The second implementation used a curve

called Catmull-Rom; it is a curve formula, where the path goes through

the control points. Also, a control point in the curve line affects only the

segment where it belongs, and not the whole curve. Catmull-Rom curves

also have easy implementation, and are fast. Closed sinusoidal shapes

designed with Catmull-Rom are diffi cult to be properly fi lled, at least

using Java fi ll algorithms; the fi ll functions have a hard time determining

which portions of the sine wave should be convex or concave to apply the

correct fi lling.

Catmull-Rom is an appropriate solution – if the target code is not supposed

to become a print piece. This kind of curve is not native to PostScript, as

the Bézier curves are. For a truly cross-media piece, the solution would be

to use the famous Bézier lines.

It’s harder to achieve the perfect variable period sine wave using Bézier;

while the Catmull implementation was effortless, this required a lot of

planning and calculation on the control points positioning. It also required

many more control points to achieve the right smooth curve visual; on the

other hand, the sketch could be the size of the screen or the size of an

ocean liner, since the vector is size and resolution independent

Finally, the pixel implementation: the best way to avoid the diffi cult fi lling

of sinusoidal shapes is to draw the pixels directly. Instead of drawing a

sine shape and fi lling it, I can set the screen pixels with different colors

according to the form of a sine wave. The pixel version is not suitable for

printing – it is limited to the 72 dpi screen dimension – but it works fi ne

on screen and despite not being optimized for speed, it runs the fi lled sine

transformations much faster than the vector versions. Another difference

105

is that while vector drawings can have anti-aliasing functions to work on

top of them, pixel manipulation can’t – the antialiasing function has to be

written specifi cally to the sketch.

The Current process exploration helps to demonstrate not only that a

visual idea can be unfolded in different ways, but also that the chosen

technique infl uences the fi nal aesthetic of the piece: the speed of motion,

the antialiasing algorithm, the curve rendering are unique in each

version. The concept is drawn on screen in a similar way, but the style of

the drawing is inherited from the coding techniques. In a computational

visual experience, the process matters.

The second part of the experiment dealt with the translation of a concept

from a gallery wall to the screen. Riley had very interesting ideas about

her painting – she says that her paintings aren’t about optical illusion,

but about optical reality. Riley explored the relationships between nature

and vision, and how the dynamic forms of nature could be represented

in their essence. Current has a very organic motion feeling into it, and I

wondered how to translate this feeling taking advantage of the computa-

tional medium.

The Flow sketch was the result of this exploration. It has two versions;

an interactive, where a user input – whether the mouse or any physical

device – triggers different wave settings and an automatic version, where

the waves oscillate by themselves. The sketch intends to capture the tran-

sition from a geometric state – the straight lines – to an organic behavior

resembling water currents. Dots were used instead of lines for more

delicacy, and also to imprint the idea that a current too strong becomes

noise; the dots mix themselves and the line pattern disappears.

Flow is a very simple sketch; it has few lines of code and uses line vertex

– no curves – and is even suitable for printing in any size. A simple

concept, simple implementation, interesting results.

Flow was also converted to an openGL version; openGL is a graphics

library that allows geometry to be calculated by the computer graphics

card, with increased performance in relation to the computer main

processor screen rendering. The Vanishing Flow mixes the renaissance

perspective with the sixties op art, creating a different aesthetic expe-

rience from the “fl at” version.

106

The op art explorations are product of my personal pursuit on exploring

and expanding this traditional art form to new possibilities; visual patterns

are great form systems that fi t naturally into the creative concepts of

computation, and my role was to make the “natural inclination” become

true.

Flow: dot and perspective versions.

109

Logic Molecules

Database: a collection of information, stored in a retrievable system.

The database concept is the foundation of the internet today; the web is

formed by uncountable servers that store all possible categories of data,

constantly retrieved by browsers worldwide.

What does a database look like?

A database has no face; physically, it is stored in a binary mode, like any

digital information - there’s no difference between a music CD and a

complex relational database source in its physical register. Logically, it

has no face either. The interface that manages the relation between user

and database has a face – but it is a “middle man”, an agent, and it can be

simple as a blinking command prompt.

The database is intangible. The nearest a human can get to it is manipu-

lating a query program, that will dive into the structured information and

return with a completed request.

Amazon, Internet Movie Database, eBay, Craigslist, Google: the major

websites serving content and services are a shell for database querying

engines – all their power lie on the amount of information they contain,

and how the information can be retrieved. How big is Amazon? How big

is eBay? Where does it begin, and where does it end? How can a database

system be understood in a graspable form?

The “Particle Database” project investigated this issue. The intent was to

connect the idea of database to a visual materiality, and raise the identity

between databases and the physical world.

The database is a mass with different categories of information in different

amounts. It has a similar structure to physical matter: a mass built by

different molecules, from different categories in also different amounts.

When a database is queried, the relevant bits of information are retrieved

by a query agent; information is added and removed of the system as well.

110

screenshots from the Particle Database querying system.

111

Building a parallel to physical matter, a database would be a fl exible, fl uid

material where the molecules could react and be attracted by different

agents.

Another feature of physical matter is density: the amount of molecules

per spatial area. A database representation would have the screen as its

total spatial area, and the density would be a key idea to represent the

amount of information populated in a system.

A potential viewer of the physical database would be able to directly

manipulate it, molding the visual data according to particular queries.

Two concepts were developed for this task; fi rst, is the compliance to the

mechanical laws of physics. Data molecules would need to be infl uenced

by gravity, and their mass generate gravitational fi elds. Acceleration and

inertia would need to be present – by refl ecting the real world, a sense

of comfort is generated in the user experience, the comfort of being ac-

quainted to physical rules like the known real world. Second, the idea of

“chemical agents”; database queries would be formulated and assume a

physical form; like a magnet or a chemical compound that would react

attracting certain types of data and repelling others. Information could

be attracted and repelled manually, allowing a sense of scale between the

amount of attracted molecules and the whole molecule universe.

The molecule/universe relationship has a clear metaphor in the compu-

tational medium: object/system. A molecule, with its components and

properties, is similar to the logical object - an element with a series of

functions and properties bound together. The concept of universe, the

multi dimensional space where molecules exist, is the same concept of a

computer system, the space where logical objects exist.

Objects are abstract data concepts; they can be modeled to contain any

visual presence on screen, and respond to any physics algorithm. Besides

an arbitrary visual representation, an object can contain actual entries

from a database.

There are scientifi c applications where a large number of objects with

properties need to be displayed and manipulated on screen; wind and

aerodynamics simulations, atomic reaction research, astronomical

models and many others. The name “particle system” is given to the style

112

of representing a large number of individual objects that have defi ned

properties, and that interact according to common rules.

The Particle Database project followed the visual style of a particle

system, having each node of information represented as a glowing pixel.

The pixels would stand grouped in a screen sized pixel sphere – the

universe, with variable density. The attraction/reaction agents would

make the particles fl ow around; other tools would allow certain particles

to be hidden, labeled or reveal their data content on screen. The project

was built as animated demo, non functional, where an example query is

done to envision the advantages and fl exibility of direct manipulation of

data.

Coded Particles

Particle systems are the most explicit example of object oriented pro-

gramming; a logical object does not need to be visual to exist – but in a

particle fi eld, each pixel is an object instance. The task of coding a particle

system consists in coding the behavior of a particle kind. This “kind” is

called class, the group of rules, characteristics and behaviors of an object.

The class describes all topics in the life of a particle – its appearance,

motion and dynamic responses for different stimulus caused by the user

or other particles.

The Database Field sketch is a study of how the Particle Database would Database Field sketch is a study of how the Particle Database would Database Field

be coded; for experimental purposes, attracting and repelling behaviors

were written, and the particles behave according to a faux Brownian

motion – random in space. The sketch adds the idea of the real interac-

tivity, absent on the application demo.

interactive particle attraction/repulsion tests

115

Patterns of Time

Edwin Abbott, English writer and clergyman, published in 1884 a short

book called “Flatland”. The book is an interesting and very early exercise

into the perception of space; the human being lives in 3+1 dimensions

– 3 spatial planes and a punctum of time. The characters of Flatland live

in a 2+1 universe; thus the fl atness of the title, the completely eliminated

perception of volume.

It is an interesting exercise to alter the relation between spatial and

temporal dimensions; the “Patterns of Time” project explored an inversion

on the perception of time and space; instead of 3+1 dimensions, 1+3. How

can we expand time in more dimensions, while compressing space to a

punctum? What is the spatial component of time?

The computational medium is a good support for this kind of experiment;

it allows great fl exibility for experimentation, and helps the imagination

in such a entangled topic. The project proposal is an interface that allows

the compression/expansion of dimensions to occur; it does not intent to

be scientifi c, but poetic.

It is easy to imagine all the special dimensions compressed to a point;

timelines do that. A timeline transforms all the physical events to a

printed point; time becomes a line, dot after dot representing day after

day, year after year. But is time a line? The characters of Abbot’s Flatland

had the fi rm belief that a third dimension didn’t exist at all – things were

fl at by nature. The visualization of time as a line of events might be stuck

in a Flatland mentality.

Pattern is a concept of many defi nitions, but they all agree in the idea that

a form may exist and be recurrent according to a rule – a confi guration.

Patterns are everywhere in nature – the confi guration of clouds, waves,

the growth of plants, structure of minerals. Common laws sew different

realms together, binding them by mathematical concepts like golden

ration and prime numbers.

opposite page: Patterns of Time Studio openGL version

116

The idea of pattern is usually employed to describe spatial or logical

events; the project idea was to extend this application to time as well

– escaping temporal Flatland.

What are the components of time?

According to a personal point of view, corroborated by the quantum

physics and approved by poetic license, time is light. Light is a periodic

wave vibration, thus time probably shares the same properties of periodic

waves - known as amplitude, wavelength and phase.

What is the amplitude, wavelength and phase of time?

Phase is the point of the amplitude at which a wave begin. Applied to

time, phase should be neutral – time starts at the beginning of time (a

recursive tautology). Amplitude and wavelength are related to the notion

of facts.

What marks the passage of time? Facts1. The happening of facts in the

physical world is a beacon for proofi ng the time fl ow.

Each fact in time is unique2; but facts share similar confi gurations. Facts

form patterns; there’s a summer in every year – it’s a predictable event.

Each summer is unique – but events that characterize the fact “summer”

happen every year.

In different scales, all facts have patterns of occurrence – some of them

are obvious, other subtle.

The facts in time are points in the amplitude wave. Each fact follows a

pattern of occurrence, so each fact has its own frequency, wavelength.

��

��

��

1 “The world is the totality of facts” – Wittgenstein, in Tractatus2 “Upon those who step into the same rivers different and ever different waters fl ow down.” - Heraclites

117

How to represent the concept of patterns in time visually?

The developed model of time used the line as a starting metaphor, added

with the pattern idea. The facts have a frequency of occurrence, so they

can have a cyclic nature. The end of each cycle does not mark the return

to the beginning; the line of time continues to the future. So the cycle is

not a closed circle – it is a line with a coil appearance.

Different patterns have different frequencies; bigger patterns encap-

sulate smaller ones – like an year encapsulate 365 days. So the coil is not

absolute – it is formed by another coil; which is formed by another and

other in an infi nite progression.

The computer is a helper to achieve such concept in a visual way; the

initial project was done after research on coil algorithms, and how could

they be applied. A recursive coil model was developed in 3D, and a Flash

demonstration of the concepts, was built around it. The “Location of

Time” is a demonstration of the concepts of time and space, and it was

projected to be the initial point of a functional interface.

Patterns of Time Studio: interactive online version

118

Patterns of Time: concept demo screens

119

The interactive version of the Location of Time was partially coded – at

the cost of much sweat – since the concept is very challenging mathemati-

cally. Infi nite levels of recursiveness in a three dimensional structure is

not exactly the everyday problem solving task of a designer. The fi rst levels

of time recursion can be explored through different points of view, and

different time line compression. Technically, the code describes a big three

dimensional polygon with variable sides, and variable vertex frequency.

The user can tweak and experiment on every possible parameter of the

timeline and the code is suitable for screen or print, in any given size.

Patterns of Time Studio: openGL version

120

121

Findings

As a result of creative process, the product of design has its structure

formed by concepts, form and medium. Concepts are ideas, consolidation

of the human imagination. The concepts become real through form (the

aesthetic faculties). And form needs the support of a medium, wherein it

can be expressed.

Noteworthy design work contains a sense of equilibrium in those com-

ponents. Such work can usually be characterized as based on strong

conceptual thinking, executed by a pleasing aesthetical form within full

employment of the medium.

We in live in a moment in time marked by the proliferation of the personal

computer and the fast growth of the internet - a computer network as a

medium. The computer, as a vast new territory for design, is offering an

interactive and dynamic space for creation.

medium

form

concepts

medium

form

concepts

time

equilibrium of the creative components

122

At this moment, the focus on computer as media seems to have outrun

the concepts in design creativity.

The focus on media innovation may have eclipsed the role of concepts

and form in creation. Instead of being based on ideas, design solutions

became based on the medium; if something can move, move it; if

something can blink, let it blink. There are innumerous examples of

senseless employment of computational strengths that confi rm Henry D.

Thoreau Arcadian quote “Our inventions are wont to be pretty toys, […]

they are but improved means to an unimproved end”.

A hundred years ago, a similar event occurred: the advent of cinema.

A new medium was introduced; a new technology was available for

expression. But there were neither an aesthetic nor language of cinema;

a new vocabulary needed to be built, where time was a plastic element

that could be manipulated through montage. The cinematic medium

component leaped ahead of available creative forms and concepts, and

only after years of worldwide experimentation a proper language of the

motion pictures emerged.

The concepts of computation should be taken and used to build a new

vocabulary of graphic design, where the computing logical plane shall be

seamlessly welded to the visual, aural, temporal and haptical dimensions.

The incorporation of computation is the key for conceptual thinking

to reclaim its leadership in design creativity, bringing reasoning and

meaning to interactive experiences.

medium

form

concepts

time

media seems to have outrun the concepts in design creativity

123

The scientifi c, mathematical approach of computation and the whimsical

nature of art are not truly opposites, and have already met numerous

times through history. We are facing a new instance of this integration,

where algorithms are agents of translation from concept to form, from

form to medium.

Designers confront the challenge of learning the praxis of computation;

creative freedom in new media can only be achieved by knowing how to

deal with the transformation machine. This knowledge does not imply

that designers should become computer scientists, but that designers

should become wiser.

The visual expression through code is not – and should not – be hype,

a trendy toy of ephemeral design. Neither should it be an obligation, a

mandatory solution to all communication questions. The visual expression

through code should be considered as a current stage of the evolution in

design which is a process where different media can coexist.

The articulation of visual experiences through algorithm is the compre-

hension of new media design as an integrated whole: when conceptual

and technical barriers are tore down, creativity can be expressed simulta-

neously in all dimensions.

Creativity is borderless.

125

Bibliography

Arnheim, Rudolph. Art And Visual PerceptionArt And Visual Perception. University of California

Press, 1974.

Ashton, Anthony. HarmonographHarmonograph. Walker, 2003.

Barthes, Roland. Camera Lucida. Hill and Wang, 1982.

Bense, Max. Estetica. Perspectiva, 1975.

CAVS-MIT. Gyorgy Kepes – The MIT yearsGyorgy Kepes – The MIT years. MIT Press, 1978.

Chediak, Almir. Harmonia & ImprovisaçãoHarmonia & Improvisação. Lumiar, 1986.

Dondis, Donis. A primer of visual literacyA primer of visual literacy. MIT Press, 1973.

Eisenstein, Sergei. The Film Form. Harcourt, 1969.

Eisenstein, Sergei. The Film Sense. Harcourt, 1969.

Gerstner, Karl. The Spirit of ColorsSpirit of Colors. MIT Press, 1981.

Gerstner, Karl. The Forms of Color. MIT Press, 1986.

Kandinsky, Wassily. Concerning the Spiritual in ArtConcerning the Spiritual in Art. Dover, 1977.

Kandinsky, Wassily. Point and Line to Plane. Dover, 1979.

Kepes, Gyorgy. Language of VisionLanguage of Vision. Paul Theobald, 1944.

Kepes, Gyorgy et al. Language + VisionLanguage + Vision. George Braziller, 1966

Kudielka, Robert. Bridget RileyBridget Riley. British Council, 1978.

Maeda, John. Maeda and Media. Thames & Hudson, 2000.

Munari, Bruno. Design Design e Comunicação Visualção Visual. Martins Fontes, 1968.

Nachmanovitch, Stephen. Free PlayFree Play. Tarcher, 1991.

Toben, Bob. Space Time and BeyondSpace Time and Beyond. Bantam, 1983.

Wardrip-Fruin, Noah and Nick Norton. The New Media Reader. MIT

Press, 2003.

Wurman, Richard Saul. Information Anxiety IInformation Anxiety I. Bantam, 1990.

127

The following is my fi rst interactive object, developed at age of 5. The procedure is simple:

1) turn off all the lights

2) close your eyes and position a camera fl ash right in front of them

3) spin your boy, a lot

4) when it’s getting almost impossible to spin, fi re the fl ash as many times as possible.

I can describe the effects: colored circles, patterns, fl oating around. There’s a scientifi c explanation, but it doesn’t really matter.

Date post:	28-Dec-2015
Category:	Documents
Upload:	clunetta
View:	21 times
Download:	0 times

The articulation of visual experiences through algorithm

Documents