Structure of Artist Virtual Environments
Mark Green
School of Creative Media
Introduction
We have examined some of the techniques for building virtual environments
time to look at some real environments: how they are structured techniques used to construct them what we need to support
then look at narrative structure and some of the other things we need
Artistic Virtual Environments
Very few virtual environments are well documented, lack of good visual and written descriptions
without good documentation can’t analyze the techniques used, not enough information
start with one project: Art and Virtual Environments, Banff Canada, 1991-94
Art and Virtual Environments
Around a dozen pieces produced over a three year period
hardware: SGI Onyx1 RE (4Mb texture, 64 Mb main
memory, R4000 processors) mainly HMD, some projection Fastrak and Dataglove for input
Bar Code Hotel: Perry Hoberman
Bar Code Hotel
Bar Code Hotel
Features: non-standard device configurations autonomous object behaviors users suggest behaviors, don’t control detailed
motion, commands like: chase, drift, wall flower and punch
richness comes from the interactions between multiple users
Archeology of a Mother TongueToni Dove and Michael Mackenzie
Takes place in a theatre, large screen projector, a guide runs the piece navigating with a Dataglove
approximately 40 minutes for the entire piece
strong emotional responses from the audience, not like a typical VR piece
Archeology of a Mother TongueToni Dove and Michael Mackenzie
Three main features: strong visual effect minimal interaction, mainly navigation strong narrative component
the piece tells a story, the guide moves the audience between the different parts of the story
Archeology of a Mother TongueToni Dove and Michael Mackenzie
Visual Richness
Narrative
Narrative structure consists of scenes and transitions
the story is told in the scenes, each scene contains part of the story
the transitions are used to move between scenes, change the setting for the next part of the story, usually short: 10 seconds
PlaceholderBrenda Laurel and Rachael Strickland
A shared environment, two participants
has a much looser narrative structure, more concentration on interaction
the interaction drives the story
intended to be an experience, participants take turns playing the four characters in the story
PlaceholderBrenda Laurel and Rachael Strickland
PlaceholderBrenda Laurel and Rachael Strickland
Features: role playing, users take on characteristics of
other characters, modify their view and behavior
voiceholders: ability to leave marks on the environment
interaction between participants spatial sound
Voiceholders
Artifacts in environment, used to record user’s voice
user grabs an empty voiceholder to record a message
grab a full voiceholder to hear the message
build up narrative by leaving messages for future users
Scenes and Transitions
Three scenes: cave, waterfall and hoodoos
each scene has a different visual presentation and characteristics, not story driven
users control scene transitions by entering a portal to another scene
10 second transition for user orientation
The Bush Soul
A more recent piece developed by Rebecca Allen at UCLA
under development for a number of years, with a good sized team
shown in 1998 and 1999
PC based multi-screen system with joystick as the main input device
The Bush Soul
This piece has a linear narrative structure, the user visits four sites over a virtual day
environment populated by creatures
these creature are autonomous, they react to each other, the user and actions that occur in the environment
user can enter the body of one of the creatures
The Bush Soul
The Bush Soul
Summary
Features that should be supported object behavior
autonomous user controlled
non-standard device configurations interaction
with user between objects between users
Summary
Features (continued): visual richness multiple media: graphics, sound, force, … scenes and transitions, support for narrative
structure control of user
how they experience the environment where they can move
More Behavior
The behavior we have done so far has been mechanical, it doesn’t appear intelligent
we will now look at ways of doing more intelligent behavior
what do we mean by intelligent behavior?
What a real organism might do?
Something that doesn’t look stupid?
More Behavior
The features of intelligent behavior: goal directed react to environment avoid collisions or other accidents social actions: move towards objects we like,
away from objects we don’t like do what we might do??
Goal Directed Behavior
Often over rated, many assume that all behaviors are goal directed
this makes the theory and programming much easier
we can select the goals, and then produce the appropriate actions
can string goals together to achieve larger goals
Goal Directed Behavior
Goals can have hierarchies
short term goals are short lived, solve an immediate problem
longer term goals take longer to reach, require more planning and time
long term goals usually divided into a sequence of shorter term goals
Goal Directed Behavior
Many AI and Alife techniques have been developed based on this approach
lots of algorithms to draw on
But, are we really goal directed?
Some of the time we definitely are, when we have a problem to solve, or something that needs to be done
Goal Directed Behavior
Other time assigning goals to our actions is quite difficult: watching cartoons on TV going for a walk browsing in a store (or the web)
we can try to assign goals to these activities, but they sound more like excuses
Goal Directed Behavior
Quite often we assign goals to people’s actions when we really have no idea of what they are doing
this makes life seem more rational
Example, a large crowd of people walking, we assume they all know where they are going, this may not be the case
Goal Directed Behavior
Doing good goal directed behavior is quite hard, often looks more like a robot than a living object
it’s not clear that it buys us anything, better off spending our time developing other behaviors
if it looks like it could have a goal, its probably good enough
Realistic Behavior
Actions should look like they have a purpose, object doesn’t do anything stupid
we want our objects to move around the environment: mainly travel in a straight line, so they look like
they know where they are going avoid running into other objects
start by looking at a simple object, mover
Mover
The mover moves in a simple environment: floor four walls
both the floor and walls are made from the same plane
want to make the mover look like it has a purpose, not just a random walk
the mover is a small cube
Mover
Two observations: we walk in the direction we are facing we walk in straight lines as long as possible
in each tick the mover moves a small amount in the direction its facing
the push action does this:push distance;
to change direction use rotz
Mover
Mover
Direction of motion
Mover
When we reach an obstacle (a wall) we need to do something
one solution is to turn so we are no longer facing the obstacle
two questions: how do we know when we are about to hit
something? How much do we turn?
Mover
The hit action can assist with obstacle avoidance
hit shoots a ray, given a starting point and direction, returns any object hit and the distance to the object:hit x, y, z, rx, ry, rz, obj;
first three parameters are start of ray, next three are the ray direction
Mover
The last parameter is a variable, name of hit object stored in this variable
the variable “distance” is set to the distance to the object
what direction and position should we use?
The position and direction of the object!
This information is stored in a set of variables
Mover
Object position: Locx, Locy, Locz
Object direction: Anglex, Angley, Anglez
hit functions as eyes, looks to see what is in front of the object
since the environment is enclosed, we will always hit something, so should we always turn?
Only if the obstacle is close
Mover
The environment is 12x12, we are moving at 6 units/second
start avoiding collision when we are 2 units from obstacle
the basic motion is done in the tick event handler
the turn event handler is used for basic obstacle avoidance
Mover
on tickeval v = 6*dt;push v;hit Locx, Locy, Locz, Anglex, Angley, Anglez, obj;select obj;if distance < 2;
trigger “mover”, “turn”;endif
end
Mover
Now we need to turn
the basic idea is to turn a bit (0.4 radian), and see if we are still going to collide
this is repeated until the collision is avoided
if for some reason the mover gets too close to the obstacle (less than 0.2 units) we have an emergency action, the back event handler
Mover
In the back event handler we move back two steps and then make a large turn (1.5 radians, about 90 degrees)
we then continue the motion
this should avoid any potential disaster
Mover on turn
spinz 0.4;hit Locx, Locy, Locz, Anglex, Angley, Anglez, obj;select obj;set flag 0;if distance < 2 ;
set flag, 1;endifselect flag;triggerif distance < 0.2, me, “back”;triggerif distance > 0.2, me, “turn”;
endon back
eval dx = -2*v;push dx;spinz -1.5;
end
Mover
Mover
What we’ve done so far gives us basically circular motion
the mover travels in a circle just inside the walls
this looks kind of stupid
periodically change the objects direction, a random change in direction, every 4 to 10 seconds
Mover
on changerandom dz, 1.20, 3.14;spinz dz;random t, 4, 10;triggerAt “mover”, “change”, t;
endon first
translate 0, 0, 0.15;triggerAt “mover”, “change”, 5;
end
Mover
Gives relatively good basic motion
can introduce a number of obstacles into the environment, a set of cubes
the mover can successfully avoid them
this gives us one motion unit, later we will add more behavior to the mover
Mover
The Thing
The mover can move around the environment, but it only reacts to obstacles
the thing is a plant-like creature that reacts to the user
the thing normally bends back and forth at a fairly slow rate (think about a plant slowly blowing in the wind)
The Thing
The Thing
The thing has a stem and two leaves, more leaves can be added by cut and paste
the leaves are separate objects with their own event handlers
the main tick event handler produces the waving motion
this is done by a rotation about the y axis
The Thingobject thing
add mesh stemadd object leaf1add object leaf2on tick
eval angle = angle+sign*da*dt;roty angle;if angle > 0.5;
eval sign = -1;endifif angle < -0.5;
eval sign = 1;endif
end
The Thing
Three variables control the motion: angle: the current rotation angle da: the rate that the angle changes sign: the direction of motion
two if actions are used to keep the angle within a reasonable range, between -0.5 and 0.5 radians
The Thing
on firstset angle, 0;set da, 0.2;set sign, 1;translate -3, 3, 0;
end
The Thing
The first event handler initializes the variables and positions the thing
the tick and first event handlers in the leaf1 and leaf2 objects are basically the same
their rotation is about the z axis
don’t need to position the leaf objects in the first event handler
The Thing
The thing reacts to the user, as the user moves closer it moves faster
when the user is within 2 units it moves 5 times faster, when the user is within 1 unit it moves 10 times faster
this is handled by a second tick event handler, the thing has two tick event handlers that operate in parallel
The Thing on tick
where “cursor”, dx, dy, dz;if distance < 2;
set da, 1.0;endifif distance < 1;
set da, 2.0;endifif distance > 2;
set da, 0.2;endifput “leaf1”, da, da;put “leaf2”, da, da;
end
The Thing
The where action is used to determine the location of a particular object:where object, dx, dy, dz;
the first parameter is the name of the object, and the next three parameters are variables that contain the distance to the object along the x, y and z axis
the distance variable is set to the distance to the object
The Thing
The cursor object represents the user, so we use where to find out how far away it is
next three if actions are used to set the rotation rate based on the distance to the user
the rotation rate for the leaves must also change, we could do the same thing in the leaf objects, but that’s a waste of time
The Thing
Since the rotation rate is the same in the leaf object, we just use the value we computed
the put action is used to change the value of a variable in another object:put object, variable, value;
the first two parameters are the object and the variable, the last parameter is the new value for the variable
Back to the Mover
What happens if we have more than one mover?
Should they interact with each other?
Should they treat other movers differently from other objects in the environment?
In the next example movers will try to move together
Movers
The basic idea is that movers want to move as a group
if there is another mover nearby, a mover will attempt to move in the same direction as that mover
it will adjust its direction of movement so its close to that of the other mover
Movers
How does this work?
First we change the mover object so none of the event handlers use the mover name, they use the me variable instead
next we duplicate the mover description two times, give the new moves slightly different names
now have three independent movers
Movers
Next each mover needs to detect other movers in the environment
they can only group together if they are close, two movers at opposite ends of the environment don’t group
we use the closest action to find the closest object to a mover, then we check to see if its another mover
Movers
The closest action has the following form:closest dx, dy, dz, object;
the first three variables get the distance to the object along the x, y and z axis, the last variable gets the name of the object
the distance variable is also set to the distance to the object
how do we detect another mover?
Movers
All mover objects have a type variable, and this variable is set to 1 (this is done in the first event handler)
we use get to get the value of this variable:get object, variable, variable;
the first two parameters are the object and variable, the last parameter gets the value
Movers
on tickclosest dx, dy, dz, obj;select obj;get obj, type, t;triggerif t == 1, me, “flock”;
endon flock
get obj, Anglez, angle;eval da = 0.3*(angle-Anglez);spinz da;
end
Movers
We use a triggerif to produce a flock event if the type is 1 (another mover)
in the flock event we get the heading of the other mover (the Anglez variable)
find the difference with our heading, scale it, and rotate in that direction
this will bring the headings together, fairly quickly
Movers
Discussion
Movers tend to pair up, sometime a group of three is formed
may get more grouping if there are more
they tend to move parallel to each other, but sometimes they follow each other
instead of changing the heading we could have moved them so they are closer to each other
Discussion
Changing heading works well in this case because they are moving fast relative to the size of the environment
could introduce other types of objects
could have a predator/prey scenario where one object chases another
instead of moving closer, prey will try to flee
Problems
If we watch this environment long enough we will notice a problem
sometimes a mover will escape from the environment
what’s happening here?
The two behaviors are interfering with each other, they have cross purposes that produce an undesirable effect
Problems
The movement and collision behavior is executed first, it changes the heading of the object so it won’t go through a wall
the grouping behavior then executes, it also changes the heading, but in the opposite direction
since it executes last, it overrides the collision avoidance behavior
Problems
Whenever two behaviors control the same variable, in this case the direction of the object, this problem can occur
when we start producing realistic behaviors this problem is inevitable
there are numerous possible solutions, we will look at a few of them
Behavior Conflicts
In this case a simple solution is to change the order of the event handlers
do grouping first, then collision avoidance
this works because collision avoidance is always more important
we can perform a strict ordering of the behaviors based on their importance
Behavior Conflicts
What happens if we can’t order the behaviors?
Either they are equally important, or in some cases one is more important, and in others the other is more important
one way of handling this is to have each behavior calculate how important it is, and only execute the most important one
Behavior Conflicts
Another approach is to allow a behavior to stop other behaviors
in our example, when a collision is close, the collision behavior stops all the other behaviors until the problem is solved
these two solutions select one of the behaviors to use, they ignore all the other behaviors
Behavior Conflicts
Another approach is to allow all the behaviors to contribute
in our case, the two behaviors produce a new direction, an angle value, the two angle values are then combined to produce the final angle
we could weight each value by the importance of the behavior
Behavior Conflicts
We could also use the high level goals of the object to determine the behavior to use, or how their outputs are weighted
but this returns us to the idea of goals
maybe this is a good use for them, as a way of avoiding conflicts
Goals, Again
We could interpret what we see in terms of goals
one of the movers gets pushed out of the environment, leaving two movers
we could interpret this as some form of mating ritual, one mover gets rid of its competition
we know this isn’t the case, but ...
Narrative Structure
Most of the artistic pieces have some structure, they are divided into scenes
the user doesn’t spend all of their time in one place, they move from place to place
each of these places has a different set of objects and different behaviors
there is usually a transition process
Narrative Structure
Narrative isn’t a new idea, its used in many media has been developed over the centuries
most media use a linear narrative structure, there is only one path through the story
the story teller guides you on this path
most media really only allow one path
Narrative Structure
Digital media introduced the idea of more complex narrative structures
this really isn’t a new idea, but most people doing digital media think it is
interactive fiction is based on a branching or tree structure
at key points in the story, the user decides which branch to follow
Narrative Structure
Besides branching, there are also loop structures and hyper links
some narratives allow branching, but bring the branches together before the next phase of the story
common in games, each level has many branches, but at the end they all join before going to the next level
Narrative Structure
We can think, or model this structure in the following way
there are two main components to our model: scenes transitions
scenes provide the location for the action, transitions move us between scenes
Narrative Structure
This is a simplified model, but it works okay for digital media
each scene has a set of active objects, the objects visible in the scene, user interacts with these objects
set of active objects changes from scene to scene, way of organizing the content
Narrative Structure
Transitions used to move from scene to scene
can be instantaneous or require some time
time can be used to: re-orient the user to a new location read in new content provide structure to the story, inform the user
that things are changing
Narrative Structure
There is some debate about non-linear narrative in digital media
interactive character development more important than branching structure
non-linear structure leads to shallow stories, many disjointed pieces
waste of time developing all of the branches
Narrative Structure in SE
The main unit of narrative structure in SE is the scene
like an object, but has no geometry
has a set of event handlers that control the overall flow of events within the scene
most scenes have entry and exit event handlers, executed when scene is entered and left
Scenes in SE
Following actions used to control set of active objects:activate object;
deactivate object;
deactivateAll;
activate makes the object active, visible and interactive
usually done in the entry event handler
Scenes in SE
Deactivate removes an object from a scene
deactivateAll deactivates all of the objects
these actions are usually used in exit event handlers
objects can be activated and deactivated at any point in a scene
works on objects, lights, sounds other scenes
Scenes in SE
If an environment has scenes, SE automatically activates the first scene and sends the entry event to it
can also specify the initial scene on the command line
can have more than one active scene at a time, not sure if this is useful
Example
Build an environment with two scenes and a transition effect
start in scene first, button press moves us to scene second
on entry there is a transition effect, more on this later
button press goes back to scene first
First scene
scene firston entry
activate “cursor”;trigger “cursor”, “first”;activate “xyz”;
endon select
triggerAt “first”, “exit”, 0.1;endon exit
deactivate “cursor”;deactivate “first”;activate “second”;trigger “second” “entry”;
endend
First Scene
On entry we activate the cursor and the spot light
when button is pressed, schedule the exit event (note same event is used for second scene transition)
the exit event deactivates all of the objects, activates the next scene and send it an entry event
Second Scenescene second
on entryactivate “cursor”;activate “test”;activate “xyz”;activate “light”;trigger “second”, “transition”;
endon exit
deactivateAll;activate “first”;trigger “first”, “entry”;
end
Second Scene
Start by activating the objects in the scene, the cursor, the sphere, the spot light and the light used for the transition effect, then trigger the transition effect
the exit event handler deactivates all of the objects, activates the first scene, and then sends and entry event to that scene
Second Scene
on selecttriggerAt “second” “exit”, 0.1;
endon transition
defaultLights 0;background 0, 0, 0;ambient 1.5, 1.5, 1.5;trigger “light”, “fadeup”;
endend
Second Scene
The select event trigger the exit from the scene
the transition event handler does the transition effect
this starts with the scene being dark, and fades up the lights over a short period of time
need to explain lighting environment
Lighting in SE
For transition effects we need to know more about how lighting is handled
SE has two default lights, an ambient and a point light
we can see things without defining a light in our scripts
if we want a fade we need to get rid of the default lights
Lighting in SE
Without default lights we can have a dark environment
the defaultLights action controls the default lighting:defaultLights flag;
if flag is zero the default lights are turned off, otherwise they are turned on
Lighting in SE
The background action can be used to change the background colour:background r, g, b;
the ambient action is used to control the ambient light level in the environment:ambient r, g, b;
the default ambient level is (0.2,0.2,0.2)
Transition Effect
The transition event handler turns off the default lights, sets the background to black and then sets a relatively high ambient light level
if then tells the light source light to fade up
this is a point light source located at the origin, its initial colour is (0,0,0), its basically turned off
Light Fade
light lightcolour 0 0 0position 0 0 0type pointon fadeup
set r, 0.02;set g, 0.02;set b, 0.02;set dr, 0.02;set dg, 0.02;set db, 0.02;triggerAt “light”, “fade”, 0.05;
end
Light Fade
The fadeup event handler initialises the variables used in the fade
the variables r, g, b are the colour of the light source
the variables dr, dg, db are the amount the colour changes each time its changed
the colour is updated 20 times per second
Light Fade
on fadeeval r = r+dr;eval g = g+dg;eval b = b+db;colour r, g, b, 1;eval r1 = r/2;eval g1 = g/2;eval b1 = b/2;background r1, g,1 b1;if r > 0.99;
set dr, 0;endifif g > 0.99;
set dg, 0;endif
if b > 0.99;set db, 0;
endifselect dr;triggerAt “light”, “fade”, 0.05;
end
Light Fade
Fade starts by computing new r,g,b colour of light and setting the light colour
the colour action is used to change the colour of a light or object:colour r, g, b, alpha;
first three components are colour, the fourth is the alpha, alpha not used for lights
Light Fade
Background also changes with light level, but at half the rate
next compute colour of background, r1, g1, and b1 and set background
next check for the end of the fade, occurs when light colour is (0.99,0.99,0.99)
Light Fade
Use if to detect when component reaches 0.99, and then set the increment to zero
this allows different fade rates and maximum values for each colour
a select on dr controls the trigger that causes the next colour update
a triggerAt with 0.05 gives 20 colour updates per second