Using movement and intentions to understand human activity Jeffrey M. Zacks * , Shawn Kumar, Richard A. Abrams, Ritesh Mehta Washington University, Psychology Department, Campus Box 1125, St. Louis, MO 63130-4899, United States article info Article history: Received 9 February 2008 Revised 11 March 2009 Accepted 18 March 2009 Keyword: Event perception abstract During perception, people segment continuous activity into discrete events. They do so in part by monitoring changes in features of an ongoing activity. Characterizing these features is important for theories of event perception and may be helpful for designing information systems. The three experiments reported here asked whether the body movements of an actor predict when viewers will perceive event boundaries. Body movements were recorded using a magnetic motion tracking system and compared with viewers’ segmenta- tion of his activity into events. Changes in movement features were strongly associated with segmentation. This was more true for fine-grained than for coarse-grained bound- aries, and was strengthened when the stimulus displays were reduced from live-action movies to simplified animations. These results suggest that movement variables play an important role in the process of segmenting activity into meaningful events, and that the influence of movement on segmentation depends on the availability of other information sources. Ó 2009 Elsevier B.V. All rights reserved. 1. Introduction Event segmentation is the process by which people break up a continuous, fluid activity into meaningful events. For example, an observer of a baseball game might perceive it as consisting of innings, at-bats, and individual pitches. For activities less structured than baseball there may not be strong norms for where the boundaries between events go; nonetheless observers often show excellent agreement in placing their boundaries (Dickman, 1963; Newtson, 1976). Neuroimaging and EEG studies suggest that event segmentation is an ongoing concomitant of normal percep- tion—people do it all the time, whether or not they are con- sciously attending to events and their boundaries (Sharp, Lee, & Donaldson, 2007; Speer, Reynolds, & Zacks, 2007; Zacks, Swallow, Vettel, & McAvoy, 2006c; Zacks et al., 2001a). The event boundaries that people identify are important for later memory. In immediate memory, one’s representa- tion of the current event appears to act as a working mem- ory buffer, with information in the buffer more accessible than comparable information from previous events (Gernsbacher, 1990; Speer & Zacks, 2005; Swallow, Zacks, & Abrams, 2009). After viewing a movie, pictures taken from event boundaries are remembered better than pic- tures taking from intervening moments (Newtson & Eng- quist, 1976). Asking viewers to attend to events at different temporal grains affects their later memory (Han- son & Hirst, 1989, 1991; Lassiter, 1988; Lassiter & Slaw, 1991; Lassiter, Stone, & Rogers, 1988). Finally, across indi- viduals, event segmentation is correlated with later mem- ory for events (Zacks, Speer, Vettel, & Jacoby, 2006a). This suggests that better understanding of event segmentation may be important for understanding and improving mem- ory. Understanding event segmentation also may be help- ful for constructing systems to automatically segment continuous data streams such as video recordings or sensor data (Mann & Jepson, 2002; Rubin & Richards, 1985; Rui & Anandan, 2000)—if one can quantitatively characterize the cognitively natural breaks between events it may be possi- ble to identify them automatically and use them to select key frames for visualization or units of analysis. 0010-0277/$ - see front matter Ó 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.cognition.2009.03.007 * Corresponding author. Tel.: +1 314 935 8454; fax: +1 314 935 7588. E-mail address: [email protected](J.M. Zacks). Cognition 112 (2009) 201–216 Contents lists available at ScienceDirect Cognition journal homepage: www.elsevier.com/locate/COGNIT
Using movement and intentions to understand human activity
Jeffrey M. Zacks *, Shawn Kumar, Richard A. Abrams, Ritesh MehtaWashington University, Psychology Department, Campus Box 1125, St. Louis, MO 63130-4899, United States
a r t i c l e i n f o
Article history:Received 9 February 2008Revised 11 March 2009Accepted 18 March 2009
Keyword:Event perception
0010-0277/$ - see front matter � 2009 Elsevier B.Vdoi:10.1016/j.cognition.2009.03.007
During perception, people segment continuous activity into discrete events. They do so inpart by monitoring changes in features of an ongoing activity. Characterizing these featuresis important for theories of event perception and may be helpful for designing informationsystems. The three experiments reported here asked whether the body movements of anactor predict when viewers will perceive event boundaries. Body movements wererecorded using a magnetic motion tracking system and compared with viewers’ segmenta-tion of his activity into events. Changes in movement features were strongly associatedwith segmentation. This was more true for fine-grained than for coarse-grained bound-aries, and was strengthened when the stimulus displays were reduced from live-actionmovies to simplified animations. These results suggest that movement variables play animportant role in the process of segmenting activity into meaningful events, and that theinfluence of movement on segmentation depends on the availability of other informationsources.
� 2009 Elsevier B.V. All rights reserved.
1. Introduction
Event segmentation is the process by which people breakup a continuous, fluid activity into meaningful events. Forexample, an observer of a baseball game might perceive itas consisting of innings, at-bats, and individual pitches. Foractivities less structured than baseball there may not bestrong norms for where the boundaries between eventsgo; nonetheless observers often show excellent agreementin placing their boundaries (Dickman, 1963; Newtson,1976). Neuroimaging and EEG studies suggest that eventsegmentation is an ongoing concomitant of normal percep-tion—people do it all the time, whether or not they are con-sciously attending to events and their boundaries (Sharp,Lee, & Donaldson, 2007; Speer, Reynolds, & Zacks, 2007;Zacks, Swallow, Vettel, & McAvoy, 2006c; Zacks et al.,2001a).
The event boundaries that people identify are importantfor later memory. In immediate memory, one’s representa-
. All rights reserved.
x: +1 314 935 7588.ks).
tion of the current event appears to act as a working mem-ory buffer, with information in the buffer more accessiblethan comparable information from previous events(Gernsbacher, 1990; Speer & Zacks, 2005; Swallow, Zacks,& Abrams, 2009). After viewing a movie, pictures takenfrom event boundaries are remembered better than pic-tures taking from intervening moments (Newtson & Eng-quist, 1976). Asking viewers to attend to events atdifferent temporal grains affects their later memory (Han-son & Hirst, 1989, 1991; Lassiter, 1988; Lassiter & Slaw,1991; Lassiter, Stone, & Rogers, 1988). Finally, across indi-viduals, event segmentation is correlated with later mem-ory for events (Zacks, Speer, Vettel, & Jacoby, 2006a). Thissuggests that better understanding of event segmentationmay be important for understanding and improving mem-ory. Understanding event segmentation also may be help-ful for constructing systems to automatically segmentcontinuous data streams such as video recordings or sensordata (Mann & Jepson, 2002; Rubin & Richards, 1985; Rui &Anandan, 2000)—if one can quantitatively characterize thecognitively natural breaks between events it may be possi-ble to identify them automatically and use them to selectkey frames for visualization or units of analysis.
202 J.M. Zacks et al. / Cognition 112 (2009) 201–216
How does the mind-brain identify event boundariesfrom the continuous stream of sensory input? Event Seg-mentation Theory (EST) (Zacks, Speer, Swallow, Braver, &Reynolds, 2007) proposes that event segmentation arisesas a side effect of ongoing understanding. To understandan ongoing event, an observer processes incoming infor-mation to generate predictions about what will happenin the near future. Such predictions allow for adaptive pro-active actions, and are a key feature of models of control inpsychology (Neisser, 1967) and neuroscience (Schultz &Dickinson, 2000). EST proposes that everyday activity in-cludes substantial sequential dependency, which can helpprediction. For example, consider watching a friend makesalad. One can make predictions about what will comenext based on conceptual features such as inferredgoals—if the friend takes out a knife this implies the goalof cutting something. One can also make predictions basedon perceptual features such as those that arise from biolog-ical motion—if the friend begins chopping with a particularfrequency and amplitude those parameters are likely to bestable. According to EST, perceivers take advantage of suchpredictability by maintaining working memory represen-tations of the current event, called event models. However,when one event ends and another begins (when the friendfinishes the salad), many of the predictive relationshipswill no longer hold. At such points one’s predictions willtend to generate more errors, and it would be adaptive toupdate one’s event models to capture the new event thathas begun. EST proposes that when prediction error in-creases transiently, comprehenders update their eventmodels. This is perceived as an event boundary. Eventboundaries are processed simultaneously on multipletimescales, a suggestion supported by physiological stud-ies (Sharp et al., 2007; Speer et al., 2007; Zacks et al.,2001a). For identifying fine-grained event boundaries, thesystem monitors prediction error over shorter time inter-vals and identifies brief transient increases; for identifyingcoarse-grained event boundaries, the system monitorslonger intervals and identifies increases that are largerand longer.
EST and other psychological accounts of event segmen-tation (Newtson, 1976; Zacks et al., 2007) argue that seg-mentation depends on the processing of featurechanges—particularly those that are not predicted. Featurechanges may be conceptual, such as changes in actor’sgoals, or perceptual, such as changes in movement pat-terns. Previous studies have provided evidence that bothsorts of feature changes are correlated with event segmen-tation. Conceptual changes predict the locations of eventboundaries when comprehenders read or hear narratives(Zacks, Speer, & Reynolds, 2009), and when they view mov-ies (Zacks, Swallow, Speer, & Maley, 2006b). Physicalchanges—particularly movement—have been studied bothqualitatively and quantitatively. One qualitative studycoded the positions of actors’ bodies at 1-s intervals, andfound that changes in body configuration were associatedwith event segmentation (Newtson, Engquist, & Bois,1977). Another study (Hard, Tversky, & Lang, 2006) useda simple animation based on the classic event perceptionwork of Heider and Simmel (1944). The animation wascoded for qualitative changes in motion, such as changes
in direction or speed of motion. Such changes were corre-lated with event segmentation.
A pair of previous studies used simple animations ofpairs of point objects moving on a white background(Zacks, 2004; Zacks et al., 2006c). Movements were charac-terized quantitatively by computing the speed and acceler-ation of each of the objects, the distance between theobjects, and their speed and acceleration relative to eachother. Viewers’ segmentation of such animations was sig-nificantly correlated with changes in these movement vari-ables (Zacks, 2004). Correlations were stronger whenparticipants segmented the activity into fine-grainedevents and less strong when they identified coarse-grainedevents. Correlations were stronger for stimuli that viewersinterpreted as depicting random motion rather than goal-directed actions. Brain activity in regions specialized formotion processing covaried both with changes in move-ment information and with changes in objects’ speed(Zacks et al., 2006c). This is consistent with the hypothesisthat comprehenders perceive event boundaries in part dueto processing changes in movement variables. However,the simple stimuli used in these experiments place limitson the conclusions they can support. Naturalistic everydayaction provides rich cues from facial expression, eye gaze,and the objects in the environment. One possibility is thatobservers may monitor movement information when thereis little else available, but for rich depictions of naturalisticactivity other cues dominate movement.
This psychological approach is consistent with work inartificial intelligence on the individuation of actions frommotion. Thibadeau (1986) described a computationalscheme for identifying event boundaries in simple anima-tions. The animations are coded to provide descriptions ofthe state of the on-screen world for each frame. Changes instates of the system correspond to changes in, for example,the position of an object. Second-order changes arechanges in first-order changes. Event boundaries aredrawn from the set of second-order changes. Thus, con-stant-velocity motion does not constitute a boundary, butacceleration does. A formal analysis provided by Rubin(1985) came to a similar conclusion. They showed thereis a class of motion transitions that can be reliably identi-fied from the two-dimensional projection of a three-dimensional motion sequence and that corresponds to psy-chological boundaries. The primitive transitions are starts,stops, and discontinuities of force. These can be composedto form 15 motion transitions (e.g., a stop and a start can becomposed into a pause). Rubin and Richards proposed thatsuch motion transitions may correspond with observers’perceptual segmentation of motion sequences. Force dis-continuities can be directly identified from object’s accel-eration, provided that mass is constant. Thus, thisanalysis converges with Thibadeau’s (1986) hypothesisthat second-order changes are important for detecting psy-chological boundaries. Subsequent work (e.g., Mann & Jep-son, 2002) has built on such formal analyses to designsystems that can segment motion sequences in video.
Thus, previous results suggest that when viewers watcheveryday activities, they perceive event boundaries in partdue to processing changes in the movement in those activ-ities. However, important questions remain. First and fore-
J.M. Zacks et al. / Cognition 112 (2009) 201–216 203
most, to this point there has been no quantitative evidencethat movement features predict how viewers segment nat-uralistic action. It is tempting to generalize from the stud-ies just reviewed that used simple animations. However,such animations are quite impoverished compared tolive-action movies of naturalistic action. Comprehensionof such animations depends solely on rigid body motioninformation. On the other hand, everyday activity is richwith information about the objects being acted upon, theexpressions and eye gaze of actors, and the nonrigid artic-ulation of the body. It would not be surprising if the pres-ence of this additional information rendered therelationship between movement information and eventsegmentation negligible. Therefore, a first important ques-tion is this: Are movement variables robustly correlatedwith event segmentation when viewing naturalistic every-day activities? The two experiments reported here askedthis question by recording movements while an actor per-formed a set of everyday activities and then asking a set ofviewers to segment those activities.
A second, related question is: If movement variables arecorrelated with event segmentation during naturalisticactivities, does removing some of the other informationthat live-action video provides change this relation? If seg-mentation depends in part on information about objects,gaze, and facial expression, then removing those cuesmight strengthen the dependence of segmentation onmovement features, and perhaps to change the nature ofthe relations between movement and segmentation.Experiments 2a and 2b investigated this possibility bycomparing segmentation of live-action videos to segmen-tation of simple animations generated from the movementinformation captured by the motion tracking system.
Third, if movement and segmentation are related whenviewing naturalistic action, does this relation vary with seg-mentation grain? Events on different timescales may becharacterized by changes in different sorts of features,ranging from physical perceptual-motor features for eventson the timescale of seconds to abstract conceptual featuresfor events on the timescale of months to years (Barker &Wright, 1954). Everyday events occupy timescales from afew seconds to tens of minutes, and there is evidence thatwithin this range of timescales different features character-ize events at different levels. More fine-grained events(with median lengths of 10–15 s) are more strongly associ-ated with specific actions on objects whereas more coarse-grained events (with median lengths of 40–60 s) are moreassociated with action contexts (Zacks, Tversky, & Iyer,2001b) or conceptual features such as goals and causes(Baldwin & Baird, 1999). Such results support an interpreta-tion that for everyday activities fine-grained events aremore perceptually determined, whereas coarse-grainedevents are more conceptually determined. Data from sim-ple animations also are consistent with this view, indicat-ing that movement variables are more strongly related tofine-grained event boundaries than coarse-grained eventboundaries (Zacks, 2004). Does this hold for naturalistic ac-tion? To answer this question, Experiment 1 manipulatedthe grain at which viewers segmented activity.
Finally, these studies investigated how conceptualinformation interacts with movement information to
determine event segmentation. Previous studies havefound that providing comprehenders with a conceptualframe for an activity before reading about it can pro-foundly affect comprehension (Bransford & Johnson,1972; George, Kutas, Martinez, & Sereno, 1999; Maguire,Frith, & Morris, 1999). Conceptual framing can affect pro-cesses that are generally thought to be fast and early incomprehension, including the resolution of lexical ambigu-ity (Wiley & Rayner, 2000). Conceptual framing has alsobeen shown to affect event comprehension and segmenta-tion (Massad, Michael, & Newtson, 1979). Does one’s con-ceptual frame affect how one processes movementinformation to extract meaning? One way of describingconceptual representations of events is in terms of sche-mata, which are structured knowledge representations oftypes of things and events that one has encountered inthe past (Rumelhart, 1980). Schemata represent typicalfeature values for a type of entity and relations amongstthose features. For example, a schema for ‘‘folding laundry”might include information about the sorts of objects thatare typically involved (e.g., clothes, baskets) and the orderin which steps are typically performed. If a viewer familiarwith laundry-folding views a movie that shows a pile ofclothes, a basket, and a person performing steps suchreaching into the basket, the viewer might well activatetheir schema for folding laundry, as well as schemata cor-responding to the different object types present, and a per-son schema.
Schema activation can have two distinct effects onongoing perception. First, active schemata provide newinformation. Activating a schema for folding laundry pro-vides information about what objects are likely to be pres-ent, what steps are likely to be performed, and in whichorder. If these additional information sources play a rolein event segmentation, then their presence might weakeneffects of movement variables on segmentation. Second,active schemata can change how information is processed,by biasing processing or modulating attention. For exam-ple, imagine the viewer sees an ambiguous, partly oc-cluded motion pattern that is consistent with the actorfolding a towel in half or wringing it out. In the contextof an active laundry-folding schema that sensory signalmight receive an interpretation consistent with folding,whereas in the context of an active kitchen-cleaning sche-ma the same signal might receive an interpretation consis-tent with wringing out. Another example: activating alaundry-folding schema might increase attention to thedistance between the hands, whereas activating a hair-combing schema might increase attention to the distancebetween the dominant hand and the head. Thus, activatinga schema for an activity may affect not just the weight gi-ven to movement information, but also how that informa-tion is processed. This would be expected to affect whichmovement features are correlated with segmentation.
Thus, conceptual framing could have two distinct ef-fects on the relations between movement and event seg-mentation: Weakening the overall strength of relationsbetween movement and segmentation, and changingwhich features of movement are correlated with segmen-tation. These two possibilities are not mutually exclusive.Experiment 2 investigated these two potential effects of
204 J.M. Zacks et al. / Cognition 112 (2009) 201–216
conceptual framing by manipulating how much informa-tion viewers had about variables other than movement be-fore they segmented an activity and while they weresegmenting.
In short, the experiments reported here aimed to an-swer four questions about the role of movement informa-tion in event segmentation. First, does movementquantitatively predict segmentation of naturalistic activ-ity? Second, does removing some of the visual informationprovided by naturalistic videos increase viewers’ depen-dence on movement features for segmentation? Third, dothe relations between movement and segmentation de-pend on the grain at which the viewer segments? Finally,do the relations between movement and segmentation de-pend on conceptual framing?
Fig. 1. Still frames taken from the laundry (top), Duplos (middle), andvideogame (bottom) movies used in Experiments 1 and 2.
2. Experiment 1
In Experiment 1, participants segmented movies ofeveryday activities performed by a single actor using aset of objects on a tabletop. The actor’s movements wererecorded with a magnetic tracking system, allowing us toanalyze the relations between movement variables andevent segmentation in naturalistic activity. Viewers seg-mented the activity at both a fine and coarse grain, allow-ing us to ask whether movement was more stronglyrelated to fine-grained than coarse-grained segmentation.
2.1. Method
2.1.1. ParticipantsTwenty-six students at Washington University (ages
18–22, 20 female) participated in partial fulfillment of acourse requirement. An additional five participants failedto complete the experiment due to computer problems(two), illness (one), or failure to follow the instructions(two).
2.1.2. StimuliParticipants watched three movies of a college-aged
man performing everyday tabletop activities—foldinglaundry (498 s), building a house from Duplos (371 s; LegoGroup, www.lego.com), and assembling a video game sys-tem (240 s). For training, the initial 180 s of a movie show-ing the man assembling a cardboard shelving unit wasused. The movies were filmed from a fixed head-heightperspective using a digital camera and reduced to320 � 240 pixel resolution for display. All movies beganand ended with several seconds of the actor sitting still.Examples of the stimuli are shown in Fig. 1. The completevideos are available at http://dcl.wustl.edu/DCL/Stimuli.html.
During filming, the actor was outfitted with three mag-netic sensors to record the position of his hands and head.The hand sensors were attached to the back of the handswith medical tape; the head sensor was attached to therear top of the head using tape and a woolen cap (seeFig. 1). The motion tracking apparatus (Flock of Birds,Ascension Technologies, Burlington VT) was controlled bya PC and synchronized with the video recording after data
acquisition. Positions were recorded at 29.27 Hz. (Duringrecording of the videogame event, the system lost signalfrom 159.9 to 170.5 s in the movie. These frames were ex-cluded from analysis.)
2.1.3. Segmentation taskParticipants segmented each movie to identify bound-
aries between events. They were told that they would bewatching movies of everyday activities and that theyshould press a button on a button box whenever, in theirjudgment, one natural and meaningful unit of activityended and another began. They were told that the actorwould be wearing sensors to track the positions of his headand hands, and that they could ignore the sensors. Each
J.M. Zacks et al. / Cognition 112 (2009) 201–216 205
participant segmented all three movies twice, once to markcoarse event boundaries and once to mark fine boundaries.For coarse segmentation they were asked to identify thelargest units of activity that were meaningful to them.For fine segmentation they were asked to identify thesmallest units of activity.
Movies were presented on a Macintosh computer(www.apple.com) with a 19 in monitor, using PsyScopesoftware (Cohen, MacWhinney, Flatt, & Provost, 1993). Re-sponses were recorded using the PsyScope button box.
2.1.4. ProcedureEach participant was given either fine-grained or
coarse-grained instructions for the segmentation task andthen trained using the 180 s practice movie. The experi-menter offered to answer any questions. The participantthen segmented the three stimulus movies. This procedurewas repeated for the other segmentation grain. Order ofsegmentation grain and movie order was counterbalancedacross participants.
2.2. Results
2.2.1. Movement analysisWe analyzed the motion tracking recordings to provide
a record of the actor’s movement over time. First, the tran-sient data collection errors were corrected by visualinspection. Next, a set of 15 variables describing the actor’smovement were calculated from the position information:
� the speed of each hand and the head,� the acceleration of each hand and the head,� the pairwise distance between each of the three tracked
points (left hand, right hand, head),� the pairwise relative speed, and� the pairwise relative acceleration.
Tim
Prop
ortio
n of
Par
ticip
ants
Seg
men
ting
0.2
0.4
0.6
0.8
0 50 100 150 200 250
Fig. 2. Proportion of participants who identified a coarse or fine event
For example, if the actor were resting his left hand on hishead and then began to move it toward the table, the pair-wise left-hand-to-head distance would increase, the pair-wise left-hand-to-head speed would become positive(indicating that distance was increasing over time), andthe pairwise left-hand-to-head acceleration would becomepositive (indicating that the rate of increase of distancewas increasing over time). As the hand reached the tablethe pairwise distance would change more slowly, the pair-wise speed would approach zero, and the pairwise acceler-ation would pass through zero, become negative, and thenreach zero again. The movement variables were thenresampled to a 1 Hz rate for comparison with the behav-ioral data, using kernel estimation with a 1-s bandwidth.
2.2.2. Relationship between movement and segmentationTo compare participants’ segmentation to the move-
ment variables, we first binned each participant’s segmen-tation data to 1 s intervals. For each interval in each moviewe then counted the number of participants who identifieda fine event boundary and the number who identified acoarse boundary, producing two time series. As can be seenin the example in Fig. 2, event boundaries were clusteredsuch that some intervals were marked as boundaries by ahigh proportion of participants and others were markedas boundaries by few participants. These measures of fre-quency of segmentation over time were then comparedto the movement variables.
One simple and intuitive measure of the strength ofrelationship between a movement variable and segmenta-tion frequency is the correlation coefficient, r. For example,a positive correlation between head speed and segmenta-tion indicates that participants tended to segment whenthe head was moving rapidly. However, the simple correla-tion presumes that the movement variables and segmenta-tion are perfectly in phase. This assumption is likely not
e (s)300 350 400 450
0.2
0.4
0.6
0.8Fine
Coarse
boundary during each 1-s interval of the laundry folding movie.
Table 1Correlations between movement variables and event segmentation after shifting the movement variables by up to 5 s to maximize the absolute value of thecorrelations. Values are means across movies (SDs in parentheses).
Note: Positive lag indicates that segmentation frequency correlated most strongly with subsequent values of a movement feature; negative lag indicatessegmentation frequency correlated most strongly with previous values of a movement feature.
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Laundry Duplos Videogame
Prop
ortio
n of
Var
ianc
e (R
2 )
Fine Coarse
Fig. 3. Movement variables accounted for substantial variance in eventsegmentation in Experiment 1, particularly for fine-grained segmenta-tion. (Error bars are 95% confidence intervals.)
1 Positive lags indicate that the effect of a change in a movement variableon segmentation is seen before the change in the movement feature itself.This at first appears paradoxical, but can occur due to autocorrelation in themovement variables.
206 J.M. Zacks et al. / Cognition 112 (2009) 201–216
warranted. For example, suppose a viewer tried deliber-ately to segment when the actor’s hands were maximallyoutstretched (i.e., at local maxima in the right hand–lefthand distance). The viewer could not know precisely whenthese maxima have occurred until after they are over. Thus,the effects of hand distance on segmentation would have atemporal lag. To account for these phase relationships wefitted cross-correlation sequences between each move-ment variable and segmentation frequency for each movie(Zacks, 2004). Coarse and fine segmentation were com-bined by dividing each frequency time series by its stan-dard deviation (to compensate for greater numbers ofboundaries in the fine segmentation condition) and sum-ming the two time series. We then calculated the cross-correlation between the summed segmentation frequencyseries and each movement variable, using lags from �5to 5 1-s bins, and noted the lag with the largest correlation.The lag and correlation were stored, and the movementvariable was shifted in time using the noted lag to maxi-mize the correlation between the movement variable andthe segmentation frequency series. The means across mov-ies of the highest correlations and optimal lags are given inTable 1. As can be seen in the table, the speed and acceler-ation of the left hand and head were consistently positivelycorrelated with segmentation, as were the distances be-tween the two hands and between each hand and the head.For all of these features the mean lags were small, between�.33 and .33. Larger lags tended to be associated withsmaller optimal correlations; this makes sense becausewhen the cross-correlation sequence has no strong maxi-mum the estimate of the optimal lag will be variable. Themost frequently occurring lag was zero (16 of 45), followedby �1 (14 of 45) and 1 (5 or 45), indicating that effects of
changes in movement variables on segmentation weregenerally seen during the same 1-s interval as the changeor during the following interval.1
With these optimally shifted movement variables inhand, we performed multiple linear regression analyses
Table 2Movement variables that were significantly correlated with event segmentation for each combination of movie and segmentation grain (p < .05 corrected formultiple comparisons across movement features).
Coarse Fine
Videogame Duplos Laundry Videogame Duplos Laundry
SpeedRight hand + + +Left hand + + + +Head + + + +
AccelerationRight hand + + + +Left hand + + +Head + + + + +
+: Relationship between movement variable and segmentation was positive.�: Relationship between movement variable and segmentation was negative.
J.M. Zacks et al. / Cognition 112 (2009) 201–216 207
to answer three questions: First, how strong was the over-all relationship between movement variables and eventsegmentation for these movies? Second, did the strengthof this relationship change with segmentation grain?Third, which particular movement variables were predic-tive of segmentation? For each movie, we fit two linearregressions, one predicting the proportion of participantswho identified a coarse boundary during each 1-s interval,and another predicting the proportion of participants whoidentified a fine boundary. The predictors for both modelswere the 15 optimally shifted movement variables. Wetook the total variance accounted for in the regression(R2) as a measure of the total strength of the relation be-tween movement and segmentation, and examined theindividual regression coefficients to characterize the signand strength of the relation for individual movementfeatures.
As can be seen in Fig. 3, movement variables were sig-nificant predictors of segmentation, accounting for 8% to33% of the variance in segmentation. For all three moviesthe relationship between movement variables and seg-mentation was substantially stronger for fine segmenta-tion than for coarse segmentation.2 Table 2 indicateswhich features were significantly correlated with event seg-mentation for each combination of movie and segmentationgrain. The most consistent predictors of segmentation werethe speed and acceleration of body parts and the distancebetween the left hand and other body parts. These relation-
2 We repeated the analyses using optimal lags computed separately forfine and coarse segmentation. The results were similar, with movementfeatures still predicting fine segmentation more strongly than coarsesegmentation.
ships were always positive, indicating that participantstended to segment when the hands and head were acceler-ating or moving quickly, and when the left hand was farfrom the right hand or head. (It is worth noting that the ac-tor was left-handed.)
2.2.3. Event unit lengthsThe lengths of the units participants identified are given
in Table 3. As can be seen in the table, participants wereable to modulate their segmentation grain as instructed,identifying larger units for coarse-grained segmentationand smaller units for fine-grained segmentation. The tablealso indicates that the units identified in the videogamemovie were generally shorter than those in the other mov-ies, particularly for coarse-grained segmentation. Thesepatterns led to significant main effects of segmentationgrain [F(1, 25) = 76.2, p < .001] and movie [F(2, 50) = 13.1,p < .001], as well as a significant interaction[F(2, 50) = 11.0, p < .001].
2.2.4. Hierarchical organizationThe degree to which viewers grouped fine events hier-
archically into coarse events was assessed using the enclo-sure measure proposed by Hard, Recchia, & Tversky(submitted for publication), and using the alignment mea-sure proposed by Zacks, Tversky, & Iyer (2001). Enclosureis calculated by comparing each coarse event boundaryto the nearest fine boundary identified by the same partic-ipant. If viewers spontaneously group fine-grained eventsinto coarse-grained events, the majority of these nearestfine boundaries should fall before the coarse boundary towhich they are closest. Therefore, enclosure is calculatedas the proportion of coarse boundaries that fall after their
Table 3Coarse and fine unit lengths in seconds as a function of movie andsegmentation grain in Experiment 1. Values are means across participants(SDs in parentheses).
208 J.M. Zacks et al. / Cognition 112 (2009) 201–216
nearest fine boundary. Enclosure scores greater than 0.5indicate hierarchical organization. The mean enclosurescore was 0.59 (SD across participants = .13), which dif-fered significantly from 0.5, t(25) = 23.7, p < .001. Therewere no significant differences in enclosure across thethree movies, F(2, 50) = .04, p = .96.
Alignment measures the degree to which coarse eventboundaries correspond to a subset of fine boundaries. Itis calculated by measuring the distance from each coarseevent boundary to its nearest fine event boundary, andcomparing those distances to that which would be ex-pected if there were no relationship between the locationsof coarse and fine boundaries. Observed distances had amean of 1.67 s (SD across participants = 1.39), whereasthe null expectation was 4.82 s (SD = 3.72 s),t(25) = �5.49, p < .001. The difference was smaller for thevideogame movie (M = 2.15 s, SD = 2.55 s) than for theDuplos movie or the laundry movie (M = 3.75 s,SD = 3.41 s, and M = 3.56 s, SD = 4.35 s, respectively), andthis effect of movie was statistically significant,F(2, 50) = 3.48, p = .04. This likely reflects a scaling effect;the videogame movie produced finer-grained segmenta-tion overall, leading to smaller actual and observeddistances.
2.3. Discussion
This experiment provided a clear answer to our firstquestion: Movement variables were robustly correlatedwith event segmentation when viewing naturalistic every-day activities. Viewers were more sensitive to the move-ments of individual body parts and the distance betweenthem than to the relative speed and acceleration of thebody parts with respect to each other. This may reflect thatbody part movements are coded with respect to a commontrunk-centered reference frame (Johansson, 1973) ratherthan in terms of effector-to-effector relations. Viewerswere more sensitive to movements of the left hand thanof the right hand. This may reflect the handedness of theactor—being left-handed, it is likely that he provided moreinformative cues with the left than the right hand. Anotherpossibility, which we think less likely, is that the left handwas processed more thoroughly because it was generallycloser to the camera in the shots we selected. In futurestudies it would be of interest to systematically vary thehandedness of actors and the orientation of the camera.
The experiment also clearly answered our second ques-tion: Movement variables were better predictors of fine-grained segmentation than coarse-grained segmentation.This replicates previous findings using simple animations(Zacks, 2004), and is consistent with the view that fine
events are particularly focused on individual actions onobjects.
3. Experiments 2a and 2b
Experiments 2a and 2b investigated the interaction be-tween perceptual and conceptual information in event seg-mentation. Specifically, they were designed to answer twoquestions: First, does removing other cues to event seg-mentation strengthen the relations between movementvariables and segmentation? Second, does one’s prior con-ceptual representation of an activity affect the ongoingprocessing of movement information?
To test both of these possibilities, we manipulated thedegree to which viewers had information about the activ-ity being performed, the objects being interacted with,and the actor’s gaze and expression. In the video condition,participants segmented movies while watching live-actionmovies as in Experiment 1. For the animation-informedcondition, we created animations of the actor’s handsand head from the motion tracking data. Participants seg-mented these animations, but before doing so viewed a40-s preview of the live-action video. Finally, participantsin the animation-uninformed condition viewed the sameanimations but without the live-action preview. In allthree conditions, observers had access to the movementsof the head and hands. The conditions were designed tovary in the additional information present. Compared tothe video condition, the animation condition was designedto deprive participants of two potential bases for segmen-tation: conceptual knowledge about the activity being per-formed (schemata) and visual features other than themotion of the head and hands. In the video conditionobservers should have been able to easily recognize theactivity being performed and thus to activate relevantevent schemata, cued by the objects present and the actor’sinteraction with those objects. They also had ongoing ac-cess to a number of visual features beyond the motion ofthe head and hands: Videos provide ongoing informationabout objects’ identities and locations, the actor’s contactwith those objects, and the actor’s facial expression andgaze. Videos also provide much information about move-ment features not captured by the animations—for exam-ple, the angular motion of the head, the relativemovements of the fingers, and the movements of the el-bows. We predicted that participants in the animation con-ditions would show stronger relations between movementand segmentation than those in the video condition, be-cause they would have access to fewer additional featuresthat might affect their segmentation.
We also hypothesized that if one’s prior conceptualiza-tion of an activity affects how movement features are pro-cessed, then the two animation groups should differ in therelation between their segmentation and movement fea-tures. The animation-informed condition was specificallydesigned to provide observers an opportunity to activaterelevant schemata (during the 40-s preview) while equat-ing the visual features present during segmentation withthe animation-uninformed condition. Because participantssegmented identical stimuli in the two animation condi-
Fig. 4. Two still frames from the laundry animation used in Experiment 2.
J.M. Zacks et al. / Cognition 112 (2009) 201–216 209
tions, any differences in their segmentation patterns wouldlikely be due to conceptual representations formed by theanimation-informed group during the preview stage. Priorconceptualization, if present, should affect both the magni-tude and nature of the relations between movement fea-tures and segmentation. First, it should change howstrongly movement features predict segmentation. Wehypothesized that if activating a schema for an activityprovided additional conceptual features that fed into eventsegmentation, this would render movement features lessstrongly related to segmentation. Second, prior conceptu-alization should change which movement features predictsegmentation, rendering features that are more schema-relevant more predictive and features that are less sche-ma-relevant less predictive.
The effect of conceptual representations on event per-ception might differ for different grains of segmentation.One possibility is that fine-grained segmentation is moreperceptually driven, whereas the grouping of fine-grainedunits into larger structures is more conceptually driven. Ifso, one would expect the animation-informed and anima-tion-uninformed groups to differ more for coarse-grainedsegmentation than for fine-grained segmentation. On theother hand, fine-grained segmentation appears to be re-lated more strongly to movement features than doescoarse-grained segmentation (Experiment 1; Zacks,2004). Therefore, if conceptual structure modulates therelationship between movement and segmentation, thendifferences between the animation-informed and anima-tion-uninformed groups might be more apparent in fine-grained segmentation. To test these possibilities, Experi-ment 2a measured fine-grained segmentation whereasExperiment 2b measured coarse-grained segmentation.
A secondary goal of these experiments was to replicatethe primary finding of Experiment 1—a robust relation be-tween movement variables and event segmentation in nat-uralistic action—using a larger stimulus set. To that end wetested participants on six everyday activities. With this lar-ger stimulus set, task fatigue and boredom were a concern,so segmentation grain was manipulated between partici-pants to reduce the session length. Fine-grained segmenta-tion was tested first, in Experiment 2a, and coarse-grainedsegmentation was tested second, in Experiment 2b.
3.1. Method
3.1.1. DesignExperiments 2a and 2b were run sequentially; however
they will be considered together as one study for mostanalyses. Viewed this way, there were two independentvariables, both manipulated between participants. Grainof segmentation was fine in Experiment 2a and coarse inExperiment 2b. Stimulus condition was manipulated be-tween participants within each experiment by randomlyassigning each participant to either the video, animation-informed, or animation-uninformed stimuli andinstructions.
3.1.2. ParticipantsFifty-four students at Washington University partici-
pated in each experiment in partial fulfillment of a course
requirement (Experiment 2a: ages 18–22, 40 female;Experiment 2b: ages 18–22, 39 female). An additionaltwo participants in Experiment 2a declined to completethe experiment and were excused; one additional partici-pant in Experiment 2b was unable to complete the proto-col due to experimenter error.
3.1.3. StimuliThe three stimulus activities from Experiment 1 were
used again in Experiment 2, but the durations of the inter-vals before and after the actor appeared were changedslightly in editing, resulting in slightly different moviedurations: 501 s for folding laundry, 380 s for building ahouse from Duplos, and 245 s for assembling a videogamesystem. In addition, three new activities were used: payingbills (388 s), improvising an abstract structure with Duplos(365 s), and making a peanut butter and jelly sandwich(332 s).
For each of the activities, an animation was constructedby rendering the left and right hands as reflective greenand red balls, respectively, and the head as a reflective blueball (see Fig. 4). The left and right hand balls were con-nected to the head ball with thin gray rods whose lengthvaried as the distance between the hands and head chan-ged. To maximize accurate depth perception, the scenewas rendered with a ground plane corresponding to thetabletop height, onto which the balls cast shadows. Theanimations were created with raster3d (http://www.bmsc.washington.edu/raster3d). Both the live-actionmovies and animations were presented at 720 � 480 reso-lution. Animations were rendered at 10 fps, which was suf-ficient to produce smooth-appearing motion; live-actionmovies were displayed at their native 29.97 fps.
In this version of the videogame stimulus, the motiontracking acquisition failure (see Experiment 1 Method) be-
Fig. 5. Movement variables accounted for substantial variance in eventsegmentation in Experiments 2a and 2b. This was stronger for finesegmentation (Experiment 2a, top) compared to coarse segmentation(Experiment 2b, bottom), and stronger for the animation conditions thanfor the video condition. (Error bars are 95% confidence intervals.)
210 J.M. Zacks et al. / Cognition 112 (2009) 201–216
gan at 163.9 s and ended at 174.6 s. During this interval theanimation showed a large red X instead of the three balls.Data from this interval and the following 70.1 s were ex-cluded from all analyses.
3.1.4. Segmentation taskParticipants performed the segmentation task as in
Experiment 1. Rather than segmenting twice to mark fineand coarse units, each participant segmented once. InExperiment 2a they were given fine-grained segmentationinstructions; in Experiment 2b they were given coarse-grained instructions.
3.1.5. ProcedureEach participant was assigned to one of the video, ani-
mation-uninformed, or animation-informed stimulus con-ditions. Participants in the video condition were trained aswere participants in Experiment 1, using the same practiceactivity. They were told that they would be viewing a mo-vie of an actor engaged in an everyday activity, and thatthe objects attached to his hands and his head are thereto record his movements. Participants in the animation-uninformed condition were told that they would be view-ing a video of the movements of three colored balls con-nected by two rods, and that they should interpret themovements as being meaningful and goal-directed. Partic-ipants in the animation-informed condition were told thatthey would be viewing animations generated from themovements of an actor’s head and hands. For the practicemovie, the animation was superimposed on the corre-sponding live-action movie, in the upper left corner, toillustrate that the rendered points corresponded to the ac-tor’s head and hands.
Participants then went on to segment the six targetmovies, with order counterbalanced across participants.For participants in the animation-informed condition, eachanimation to be segmented was preceded by 40 s of thecorresponding live-action movie. This was done to maxi-mize the degree to which viewers would be able to forma vivid image of the actions performed from the move-ments of the balls in the animation.
3.2. Results
For all analyses, data from the first 40 s of each moviewere excluded, because participants in the animation-in-formed condition had previously viewed the live-action vi-deo corresponding to those 40 s.
3.2.1. Relationship between movement and segmentationMovement information was analyzed using the same
procedures as for Experiment 1. The movement informa-tion was captured, filtered, and resampled to a 1-s sam-pling rate. The segmentation data were binned to 1 sintervals and the segmentation counts for each intervalwere averaged (separately for each group) to estimatethe frequency of segmentation over time.
For each movement feature we calculated the cross-cor-relation between that feature and the combined segmenta-tion data for the three groups; this was performedseparately for each feature and each movie, and separately
for Experiments 2a and 2b. As can be seen in Table 1, cor-relations were overall higher than those in Experiment 1.This probably reflects the fact that with a larger numberof participants estimates of segmentation are more reli-able. The pattern of correlation across features was quitesimilar to that in Experiment 1: Speed and accelerationwere again strong predictors, particularly those of the lefthand, followed by the distances between the left handand the head and between the left and right hands. As inExperiment 1, the most frequently occurring best-fittinglag was 0 (59 of 90 in Experiment 2a, 50 of 90 in Experi-ment 2b), followed by 1 (14 of 90 in Experiment 2a, 10of 90 in Experiment 2b) and �1 (8 of 90 in Experiment2a, 6 of 90 in Experiment 2b). Larger lags occurred mostlywith small (i.e., unreliable) correlations.
As for Experiment 1, we performed multiple linearregression analyses with segmentation frequency as thedependent measure and the 15 movement variables asthe predictors. Regressions were performed separately foreach movie and each group. As can be seen in Fig. 5, move-ment variables were again strong predictors of segmenta-tion frequency. Replicating Experiment 1, movementpredicted fine segmentation (Experiment 2a) more
J.M. Zacks et al. / Cognition 112 (2009) 201–216 211
strongly than coarse segmentation (Experiment 2b), lead-ing to higher R2 values. To test whether the difference inR2 between coarse and fine segmentation was statisticallyrobust, and to assess whether R2 differed across the stimu-lus conditions, we conducted an analysis of variance (ANO-VA) with the R2 value from each regression as thedependent measure, grain and stimulus condition as theindependent measures, and movie as a blocking variable.The difference between fine and coarse grains was statisti-cally significant, F(1, 25) = 122.6, p < .001. Movement alsopredicted segmentation more strongly in the two anima-tion conditions than in the video conditions,F(2, 25) = 15.4, p < .001. However, the effect of stimulus
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Rig
ht H
and
Left
Han
d
Hea
d
Rig
ht H
and
Left
Han
d
Hea
d
Rig
ht-L
eft
*
*
*
*
* *
Fine
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Rig
ht H
and
Left
Han
d
Hea
d
Rig
ht H
and
Left
Han
d
Hea
d
Rig
ht-L
eft
Speed Acceleration Distan
*
Coarse
Cor
rela
tion
with
Seg
men
tatio
n Fr
eque
ncy
Fig. 6. Speed, acceleration and distance were positively correlated with segmentaare standard errors of the mean. Asterisks mark features for which there wasfeatures.)
condition was qualified by a marginally significant grain-by-stimulus condition interaction, F(2, 25) = 3.10, p = .06.To follow this up we conducted ANOVAs separately forExperiments 2a and 2b. For Experiment 2a there was a sig-nificant effect of stimulus condition, F(2, 10) = 16.3,p < .001, whereas for Experiment 2b this effect was not sig-nificant, F(2, 10) = 2.0, p = .19. To test the specific hypothe-sis that viewers’ conceptual frames affected the strength ofthe relationship between movement and segmentation, wecompared the R2 statistics for the two animation groups ineach experiment. In neither case was this difference statis-tically significant [Experiment 2a: t(5) = 0.74, p = .87;Experiment 2b: t(5) = 0.87, p = .43].
Rig
ht-H
ead
Left-
Hea
d
Rig
ht-L
eft
Rig
ht-H
ead
Left-
Hea
d
Rig
ht-L
eft
Rig
ht-H
ead
Left-
Hea
d
Video Animation Informed Animation Uninformed
Rig
ht-H
ead
Left-
Hea
d
Rig
ht-L
eft
Rig
ht-H
ead
Left-
Hea
d
Rig
ht-L
eft
Rig
ht-H
ead
Left-
Hea
d
ce Relative Speed Relative
Acceleration
tion in Experiment 2, particularly for the animation conditions. (Error barsa significant group difference, corrected for multiple comparisons across
212 J.M. Zacks et al. / Cognition 112 (2009) 201–216
To assess the strength of the relationship between indi-vidual movement features and segmentation, we com-puted correlations between each movement featureseparately for each movie for each group in each of thetwo experiments. The results are plotted in Fig. 6. Similarto Experiment 1, the speed and acceleration of body parts,and the distances between them, were the best predictorsof segmentation. Correlations for speed and accelerationagain were positive, indicating that viewers segmentedwhen body parts were moving quickly and speeding up.Correlations for distances generally again were positive,indicating that viewers segmented when body parts werefar apart. Similar groups of features were related to seg-mentation across experimental conditions. To test thesestatistically, we converted the correlations to normally dis-tributed variables with Fisher’s z transformation and sub-jected them to one group t-tests for each movementvariable, with a Bonferroni correction across the 15 fea-tures for each of the two experiments. The speed and accel-eration of the hands and head were significantly correlatedwith segmentation [Experiment 2a: smallest t(35) = 8.36,corrected p < .001; Experiment 2b: smallest t(35) = 9.71,corrected p < .001]. The distance between the head andeach hand, and the distance between the hands, were allsignificantly correlated with segmentation, [Experiment2a: smallest t(35) = 5.07, corrected p < .001; Experiment2b: smallest t(35) = 3.51, corrected p = .001]. Finally, therelative speed of the two hands, and their relative acceler-ation, both were negatively correlated with segmentation.The correlation for relative speed was not significant forExperiment 2a [t35) = �2.06, corrected p = .83] but wassignificant for Experiment 2b [t(35) = �3.55, correctedp = 0.001. The correlation for relative acceleration was sig-nificant for both experiments [Experiment 2a:t(35) = �5.18, corrected p = .001; Experiment 2b:t(35) = �5.45, corrected p < .001].
To test whether stimulus condition affected howstrongly individual movement variables correlated withevent segmentation, we conducted a set of univariate AN-OVAs with stimulus condition as the independent measureand movie as a blocking variable. This analysis was con-ducted separately for the fine (Experiment 2a) and coarse
Table 4Coarse and fine unit lengths in seconds as a function of movie and segmentation gparentheses).
Video
Fine segmentation (Experiment 2a)Bills 12.84 (8.08)Laundry 14.34 (9.04)Duplos house 9.77 (8.66)Duplos improvisation 11.41 (9.96)Peanut butter and jelly 9.29 (5.48)Videogame 8.23 (4.45)
(Experiment 2b) segmentation groups. For each ANOVAthe dependent measure was the Fisher-transformed corre-lation between a movement variable and segmentationfrequency. The results are indicated with asterisks inFig. 6. As can be seen in the figure, for fine segmentationstimulus condition significantly affected the strength ofthe correlation between segmentation frequency and sixvariables: the speed and acceleration of the left and righthands and the head. In all cases, follow-up t-tests indicatedthat the video group had a significantly smaller correlationthan one or both of the animation groups. There were nogroup differences in the correlations of segmentation fre-quency with distances between body parts, or with relativespeed or acceleration of body parts. For coarse segmenta-tion the only effect of stimulus condition on the correla-tions was for the speed of the head. Follow-up t-testsindicated that the two animation groups had significantlyhigher correlations than the video group.
In sum, movement variables were again strongly corre-lated with event segmentation. Movement was morestrongly related to segmentation for those who segmentedat a fine grain, particularly when they watched animationsrather than live-action video. Participants were more likelyto identify event boundaries (a) when the head and handswere moving quickly, (b) when the speed of the head andhands was increasing, (c) when the head and hands werefar apart, (d) when the hands were moving toward eachother, and (e) when the hands were accelerating towardeach other. For neither fine nor coarse segmentation wasthere evidence that the two animation groups differedfrom each other in the strength of the relationship betweenmovement and segmentation or in which movement fea-tures predicted segmentation. Therefore, there was no evi-dence for an effect of conceptual frame on segmentation.
3.2.2. Event unit lengthsAs can be seen in Table 4, unit lengths in Experiments
2a were comparable to those from the fine segmentationcondition in Experiment 1, and unit lengths from Experi-ment 2b were comparable to the coarse condition in Exper-iment 1. A mixed ANOVA with segmentation grain andstimulus condition as between-participants variables and
rain in Experiments 2a and 2b. Values are means across participants (SDs in
J.M. Zacks et al. / Cognition 112 (2009) 201–216 213
movie as a repeated measure indicated that coarse unitswere significantly larger than fine units, as expected,F(1, 102) = 70.51, p < .001. The video group identified largerunits than the two animation groups, particularly forcoarse segmentation, leading to a significant main effectof stimulus condition [F(2, 102) = 15.19, p < .001] and a sig-nificant grain-by-stimulus condition interaction[F(2, 102) = 10.41, p < .001]. As in Experiment 1, unitlengths varied across the movies: There was a significantmain effect of movie and all the interactions involving mo-vie were significant, indicating that the grain and stimuluscondition effects varied across the movies (smallestF = 3.44, p < .001). To better characterize the grain-by-stimulus condition interaction, we evaluated the main ef-fect of stimulus condition in separate follow-up ANOVAsfor each of the two experiments. For fine segmentation(Experiment 2a), there was no significant effect of stimuluscondition, F(2, 51) = 1.51, p = .23. For coarse segmentation,the effect of stimulus condition was significant,F(2, 51) = 13.93, p < .001. In sum, participants identifiedsomewhat larger event units from videos than from anima-tions. This was particularly true when they identifiedcoarse-grained events.
3.3. Discussion
3.3.1. Experiments 2a and 2b provided a robust replication ofthe primary result of Experiment 1
Movement variables were strongly correlated with seg-mentation of naturalistic everyday activities. This experi-ment also provided a clear answer to our question as towhether movement would be more strongly related to seg-mentation when other cues to segmentation were re-moved: The two groups who viewed relativelyimpoverished animations showed stronger correlations be-tween movement variables and segmentation than did thegroup who viewed the live-action videos. We interpret thisas indicating that the segmentation of the video groups de-pended on movement features as in the animation condi-tions, and also on the additional features that videoprovides. This may include information about objects’identities and locations, information about the actor suchas his facial expression and gaze. It also may include infor-mation about movement that is not available in the anima-tions. We also note that participants in the video groupsproduced coarser units than those in the two animationgroups. Given that coarse segmentation is associated withweaker correlations between segmentation and move-ment, it is possible that the video group’s weaker correla-tions were caused by their coarser segmentation. Infuture work it will be important to explore this further,for example by constraining segmentation grain moretightly.
This experiment gave a surprising answer to our finalquestion: Does one’s prior conceptual representation ofan activity affect the ongoing processing of movementinformation? If conceptual information such as that pro-vided by an event schema allows viewers to establish aconceptual frame to integrate perceptual information dur-ing viewing, one would expect that providing a 40-s pre-view of an activity would be sufficient to establish such a
frame in the animation-informed condition. If so, thenthe animation-informed groups should have differed fromthe animation-uninformed groups either in the strengthwith which movement features were related to segmenta-tion, or in which particular features were correlated withsegmentation. However, in neither experiment did we findevidence for such effects. Of course, this is a null result andas such it should be interpreted with caution. However, thedifferences between the animation groups and the videogroup indicate that this design had sufficient power to de-tect effects of the stimulus condition manipulation.
One account of this null result (suggested by an anony-mous reviewer) is that for the impoverished animations,viewers might attend especially carefully to movementfeatures, attempting to fit the movement information intowhatever conceptual frame had been established. If so,then one would expect that activating a schema for anactivity would preserve or even strengthen the degree towhich movement features predict segmentation. However,on this interpretation one would expect that conceptualframing would affect how movement features relate toevent segmentation, even if it does not affect how strongly.If this were the case one would expect that the previewmanipulation would have affected the sign or magnitudeof the correlations between individual movement featuresand segmentation; Fig. 6 shows this was clearly not thecase.
Another possibility is that the 40-s previews providedin this experiment did not constrain viewers’ conceptualrepresentations enough to affect segmentation such thatour measures could capture these effects. This could occurif viewers formed conceptual representations during thepreviews but then failed to retain them in memorythroughout the animation. It also could occur if viewers’conceptual representations were not sufficient to tie themovement in the animations back to the objects and goalsestablished in the preview. To investigate this, we fit linearmodels for the two animation groups using only the datafrom the first 40 s of each movie—the intervals that hadpreviously been excluded from analysis because the ani-mation-informed group previewed them prior to segmen-tation. If effects of conceptual framing on segmentationare present they should be especially strong for those first40 s, because viewers will have just seen the previews andwill have direct knowledge of how the movements in videorelate to the objects in the scene. This analysis has the po-tential to overestimate effects of conceptual framing, be-cause the animation-informed groups not only have beengiven a conceptual frame but also have seen the specificactions depicted as full-motion videos; this is why the first40 s was excluded from the main analyses. Despite the po-tential to overestimate effects of conceptual framing, weobserved no significant differences between the anima-tion-informed and animation-uniformed conditions in thestrength of the relationship between movement featuresand segmentation. There also were no significant differ-ences in the correlations between individual movementfeatures and fine segmentation. For coarse segmentation,three movement features showed small reductions in theircorrelations with segmentation that were significant at the.05 level but failed to survive corrections for multiple com-
214 J.M. Zacks et al. / Cognition 112 (2009) 201–216
parisons. In short, even in an analysis biased in favor offinding effects of conceptual framing on segmentation,there was little evidence for such effects.
4. General discussion
These experiments gave clear answers to three of thefour questions we set out to answer. First, movement vari-ables were significant predictors of event segmentation innaturalistic everyday activities. Second, when othersources of information for perceptual prediction were re-moved by reducing the live-action movies to simple ani-mations, relations between movement and segmentationwere strengthened. Third, relations between movementand segmentation were stronger for fine-grained segmen-tation than for coarse-grained segmentation.
These results are consistent with the account of eventsegmentation provided by EST (Zacks et al., 2007), whichholds that event boundaries are detected in virtue of pro-cessing unpredicted feature changes. However, correla-tions between movement changes and eventsegmentation also may be predicted by other theories ofevent segmentation. For example, one account of eventsegmentation holds that the behavior stream consists ofa spike structure in which brief bursts of change in featuresmonitored by an observer form ‘‘points of definition”(Newtson, Hairfield, Bloomingdale, & Cutino, 1987). Achange in one or more movement features could be de-fined to be such a burst. However, what determines whichfeature changes form spikes and what it means to be apoint of definition are underspecified in this account. Todiscriminate between prediction error-based accountsand other ones it would be valuable to obtain more directmeasures of prediction and prediction accuracy over time.This could be done using behavioral paradigms with an ex-plicit predictive response, such as a visuomotor trackingtask. Prediction errors also could be investigated withoutan explicit task by measuring physiological correlates suchas electroencephalographic responses associated with er-rors (Holroyd & Coles, 2002).
The finding that movement features were more stronglycorrelated with fine segmentation than with coarse seg-mentation replicates previous results (Hard et al., 2006;Zacks, 2004). This pattern supports the proposal that finesegmentation depends more strongly than coarse segmen-tation on the processing of movement features, whereascoarse segmentation may depend more strongly on con-ceptual features (Zacks & Tversky, 2001). However, the factthat coarse segmentation of animations was not affectedby the manipulation of prior information (Experiment 2b)offers a hint about the sort of information that may drivecoarse segmentation: It suggests that the features drivingcoarse segmentation may not be particularly susceptibleto top-down influences. In future research it would bevaluable to test this hypothesis using other manipulationsof top-down processing.
The finding that movement features predict when view-ers will segment activity may have applications in the de-sign of information systems. In domains such as medicaldata analysis (Christoffersen, Woods, & Blike, 2007), video
surveillance (Chellappa, Cuntoor, Joo, Subrahmanian, &Turaga, 2008) and film production (Murch, 2001) it isimportant to segment a continuous data stream into psy-chologically meaningful events. Research in artificial intel-ligence and human-computer interaction has addressedthis problem with some success (Chellappa et al., 2008;Davis, 1995; Rui & Anandan, 2000). Systems that usemovement cues to define boundaries in data streams inthe same way that people use those cues may producemore human-usable segmentations. Of course, in many do-mains it is impractical to invasively record actors’ bodymovements as was done here. However, improvementsin computer vision algorithms for biological motion recov-ery may render invasive motion tracking unnecessary(Sidenbladh, Black, & Fleet, 2000; Zhang & Troje, 2007).Automatic segmentation of activity into psychologicallymeaningful units would be helpful for producing visualsummaries of data streams such as storyboards. Automaticsegmentation also may be valuable as preprocessing formodels designed to recognize actors, actions, or affectbased on movement patterns (Pollick, Lestou, Ryu, & Cho,2002; Troje, Westhoff, & Lavrov, 2005).
The final question addressed by these experiments was:Does one’s prior conceptual representation of an ongoingactivity affect the ongoing processing of movement infor-mation? We saw little evidence for effects of one’s priorconceptual representation on movement processing. It ispossible that we lacked detection power or that our analy-ses failed to quantify the movement variables affected bythe experimental manipulation, or it could reflect a truenull result. If so, it is at first blush a somewhat surprisingresult. In studies of the comprehension of texts (Bransford,Barclay, & Franks, 1972) and movies (Massad et al., 1979),providing a prior conceptual frame has consistently beenfound to have large effects on comprehension and latermemory. The standard account of such phenomena is thatproviding a conceptual frame, say, by giving an informativetitle before reading a story, allows the reader to activatesemantic knowledge structures that facilitate integrationof incoming information. These knowledge structures, usu-ally referred to as event schemata, capture informationabout actors, actions, objects, and their relations. One pos-sibility is that typical event schemata represent informa-tion at a temporal grain coarser even than the coarsegrain studied in Experiment 2b. The events in that studycorresponded approximately to actions at the level of‘‘spreading peanut butter on bread” or ‘‘folding a pile ofshirts.” It is possible that effects of conceptual framing onsegmentation would be observed not at the level at which‘‘spreading peanut butter” fits into ‘‘making a peanut but-ter sandwich” but at the level at which ‘‘making a peanutbutter sandwich” fits into ‘‘packing lunch.” However, thisseems unlikely on its face. The temporal scale of the coarsegrain measured in this study corresponds well to some ofthe components of event schemata measured in normativestudies (Galambos, 1983; Rosen, Caplan, Sheesley, Rodri-guez, & Grafman, 2003), and to units that have been iden-tified with goal units and causal units in understandingeveryday human behavior (Bower, 1982; Cooper & Shallice,2006; Magliano, Taylor, & Kim, 2005; Trabasso & Stein,1994; Zwaan & Radvansky, 1998). Another possibility is
J.M. Zacks et al. / Cognition 112 (2009) 201–216 215
that conceptual framing affects how perceptual details areconsolidated into long term memory or affects the recon-struction of events during retrieval without affecting thesegmentation of activity into events on line. Effects ofschemata on event memory sometimes reflect differencesin reconstruction at retrieval time rather than effects onencoding processes (Anderson & Pichert, 1978; Thorndyke& Hayes-Roth, 1979). The present failure to find effects ofprior conceptualization on event segmentation is consis-tent with such accounts. Whether one of these interpreta-tions should be adopted awaits confirmation and extensionof this intriguing null result.
Heraclitus wrote that ‘‘you can never step in the sameriver twice,” in part because the river is no longer the same.This is an apt metaphor for the dynamic, fluid, and ever-changing stream of behavior. Given the complexity andvariety of everyday activity and the fact that no previousevent ever repeats perfectly, humans’ ability to navigateour environment is really quite impressive. One way peo-ple cope with the complexity and dynamics of everydayactivity is by segmenting it into meaningful chunks. Thefact that some sequences of movement features are pre-dictable appears to allow perceivers to pick out currentsin the larger stream that form meaningful events.
Acknowledgements
This research was supported by the National ScienceFoundation (0236651). The authors would like to thankBecky Hedden and Jeff Stevenson for assistance carryingout the experiments.
References
Anderson, R. C., & Pichert, J. W. (1978). Recall of previously unrecallableinformation following a shift in perspective. Journal of Verbal Learningand Verbal Behavior, 17(1), 1–12.
Baldwin, D. A., & Baird, J. A. (1999). Action analysis: A gateway tointentional inference. In P. Rochat (Ed.), Early social cognition(pp. 215–240). Hillsdale, NJ: Lawrence Erlbaum Associates.
Barker, R. G., & Wright, H. F. (1954). Midwest and its children: Thepsychological ecology of an American town. Evanston, Illinois: Row,Peterson and Company.
Bower, G. (1982). Plans and goals in understanding episodes. In A.Flammer & W. Kintsch (Eds.), Discourse processing (pp. 2–15).Amsterdam: North-Holland Publishing Company.
Bransford, J. D., Barclay, J. R., & Franks, J. J. (1972). Sentence memory: Aconstructive versus interpretive approach. Cognitive Psychology, 3(2),193–209.
Bransford, J. D., & Johnson, M. K. (1972). Contextual prerequisites forunderstanding: Some investigations of comprehension and recall.Journal of Verbal Learning and Verbal Behavior, 11(6), 717–726.
Chellappa, R., Cuntoor, N. P., Joo, S.-W., Subrahmanian, V. S., & Turaga, P.(2008). Computational vision approaches for event modeling. In T. F.Shipley & J. M. Zacks (Eds.), Understanding events: From perception toaction (pp. 279–341). Oxford: Oxford University Press.
Christoffersen, K., Woods, D. D., & Blike, G. T. (2007). Discovering theevents expert practitioners extract from dynamic data streams: ThemUMP technique. Cognition, Technology, and Work, 9, 81–98.
Cohen, J. D., MacWhinney, B., Flatt, M., & Provost, J. (1993). PsyScope: Aninteractive graphic system for designing and controlling experimentsin the psychology laboratory using Macintosh computers. BehaviorResearch Methods, Instruments and Computers, 25(2), 257–271.
Cooper, R. P., & Shallice, T. (2006). Hierarchical schemas and goals in thecontrol of sequential behavior. Psychological Review, 113(4), 887–916[discussion 917-31].
Davis, M. E. (1995). Media streams: Representing video for retrieval andrepurposing.
Dickman, H. R. (1963). The perception of behavioral units. In R. G. Barker(Ed.), The stream of behavior (pp. 23–41). New York: Appleton-Century-Crofts.
Galambos, J. A. (1983). Normative studies of six characteristics of ourknowledge of common activities. Behavior Research Methods andInstrumentation, 15(3), 327–340.
George, M. St., Kutas, M., Martinez, A., & Sereno, M. I. (1999). Semanticintegration in reading: Engagement of the right hemisphere duringdiscourse processing. Brain, 122(Pt 7), 1317–1325.
Gernsbacher, M. A. (1990). Language comprehension as structure building.Hillsdale: L. Erlbaum.
Hanson, C., & Hirst, W. (1989). On the representation of events: A study oforientation, recall, and recognition. Journal of Experimental Psychology:General, 118(2), 136–147.
Hanson, C., & Hirst, W. (1991). Recognizing differences in recognitiontasks: A reply to Lassiter and Slaw. Journal of Experimental Psychology:General, 120(2), 211–212.
Hard, B. M., Recchia, G., & Tversky, B. (submitted for publication). Theshape of action.
Hard, B. M., Tversky, B., & Lang, D. (2006). Making sense of abstractevents: Building event schemas. Memory and Cognition, 34(6),1221–1235.
Heider, F., & Simmel, M. (1944). An experimental study of apparentbehavior. American Journal of Psychology, 57, 243–259.
Holroyd, C. B., & Coles, M. G. (2002). The neural basis of human errorprocessing: Reinforcement learning, dopamine, and the error-relatednegativity. Psychological Review, 109(4), 679–709.
Johansson, G. (1973). Visual perception of biological motion and a modelfor its analysis. Perception and Psychophysics, 14, 201–211.
Lassiter, G. D. (1988). Behavior perception, affect, and memory. SocialCognition, 6(2), 150–176.
Lassiter, G. D., & Slaw, R. D. (1991). The unitization and memory of events.Journal of Experimental Psychology: General, 120(1), 80–82.
Lassiter, G. D., Stone, J. I., & Rogers, S. L. (1988). Memorial consequences ofvariation in behavior perception. Journal of Experimental SocialPsychology, 24(3), 222–239.
Magliano, J. P., Taylor, H. A., & Kim, H. J. (2005). When goals collide:Monitoring the goals of multiple characters. Memory and Cognition,33(8), 1357–1367.
Maguire, E. A., Frith, C. D., & Morris, R. G. (1999). The functionalneuroanatomy of comprehension and memory: The importance ofprior knowledge. Brain, 122(Pt 10), 1839–1850.
Mann, R. & Jepson, A. D. (2002). Detection and classification of motionboundaries. In Paper presented at the eighteenth national conference onartificial intelligence. Edmonton, Canada.
Massad, C. M., Michael, H., & Newtson, D. (1979). Selective perception ofevents. Journal of Experimental Social Psychology, 15(6), 513–532.
Murch, W. (2001). In the blink of an eye: A perspective on film editing (2nded.). Los Angeles: Silman-James Press.
Neisser, U. (1967). Cognitive psychology. New York: Appleton-Century-Crofts.
Newtson, D. (1976). Foundations of attribution: The perception ofongoing behavior. In J. H. Harvey, W. J. Ickes, & R. F. Kidd (Eds.),New directions in attribution research (pp. 223–248). Hillsdale, NewJersey: Lawrence Erlbaum Associates.
Newtson, D., & Engquist, G. (1976). The perceptual organization ofongoing behavior. Journal of Experimental Social Psychology, 12,436–450.
Newtson, D., Engquist, G., & Bois, J. (1977). The objective basis of behaviorunits. Journal of Personality and Social Psychology, 35(12), 847–862.
Newtson, D., Hairfield, J., Bloomingdale, J., & Cutino, S. (1987). Thestructure of action and interaction. Social Cognition, 5(3), 191–237.
Pollick, F. E., Lestou, V., Ryu, J., & Cho, S. B. (2002). Estimating theefficiency of recognizing gender and affect from biological motion.Vision Research, 42(20), 2345–2355.
Rosen, V. M., Caplan, L., Sheesley, L., Rodriguez, R., & Grafman, J. (2003). Anexamination of daily activities and their scripts across the adultlifespan. Behavioral Research Methods and Computers, 35(1), 32–48.
Rubin, J. M. & Richards, W. A. (1985). Boundaries of visual motion. AIM-835.
Rui, Y. & Anandan, P. (2000). Segmenting visual actions based on spatio-temporal motion patterns.In Paper presented at the IEEE computersociety conference on computer vision and pattern recognition, HiltonHead, SC.
Rumelhart, D. E. (1980). Schemata: The building blocks of cognition. In R.J. Spiro, B. C. Bruce, & W. F. Brewer (Eds.), Theoretical issues in readingcomprehension: Perspectives from cognitive psychology, linguistics,artificial intelligence, and education (pp. 33–58). Hillsdale, NJ: L.Erlbaum Associates.
216 J.M. Zacks et al. / Cognition 112 (2009) 201–216
Schultz, W., & Dickinson, A. (2000). Neuronal coding of prediction errors.Annual Review of Neuroscience, 23, 473–500.
Sharp, R. M., Lee, J., & Donaldson, D. I. (2007). Electrophysiologicalcorrelates of event segmentation: How does the human mind processongoing activity? Paper presented at the Annual Meeting of theCognitive Neuroscience Society, New York.
Sidenbladh, H., Black, M. J., & Fleet, D. J. (2000). Stochastic tracking of 3Dhuman figures using 2D image motion. In D. Vernon (Ed.), Proceedingsof the European conference on computer vision (pp. 702–718). Berlin:Springer Verlag.
Speer, N. K., Reynolds, J. R., & Zacks, J. M. (2007). Human brain activitytime-locked to narrative event boundaries. Psychological Science,18(5), 449–455.
Speer, N. K., & Zacks, J. M. (2005). Temporal changes as event boundaries:Processing and memory consequences of narrative time shifts. Journalof Memory and Language, 53, 125–140.
Swallow, K. M., Zacks, J. M., & Abrams, R. A. (2009). Event boundaries inperception affect memory encoding and updating. Journal ofExperimental Psychology: General, 138, 236–257.
Thibadeau, R. (1986). Artificial perception of actions. Cognitive Science,10(2), 117–149.
Thorndyke, P. W., & Hayes-Roth, B. (1979). The use of schemata in theacquisition and transfer of knowledge. Cognitive Psychology, 11(1),82–106.
Trabasso, T., & Stein, N. L. (1994). Using goal-plan knowledge to merge thepast with the present and the future in narrating events on line TheJohn D. and Catherine T. MacArthur Foundation series on mentalhealth and development. In M. Haith (Ed.), The development offuture-oriented processes (pp. 323–349). Chicago: University ofChicago Press.
Troje, N. F., Westhoff, C., & Lavrov, M. (2005). Person identification frombiological motion: Effects of structural and kinematic cues. Perceptionand Psychophysics, 67(4), 667–675.
Wiley, J., & Rayner, K. (2000). Effects of titles on the processing of text andlexically ambiguous words: Evidence from eye movements. Memoryand Cognition, 28(6), 1011–1021.
Zacks, J. M. (2004). Using movement and intentions to understand simpleevents. Cognitive Science, 28(6), 979–1008.
Zacks, J. M., Braver, T. S., Sheridan, M. A., Donaldson, D. I., Snyder, A. Z.,Ollinger, J. M., et al. (2001a). Human brain activity time-locked toperceptual event boundaries. Nature Neuroscience, 4(6), 651–655.
Zacks, J. M., Speer, N. K., & Reynolds, J. R. (2009). Situation changes predictthe perception of event boundaries, reading time, and perceivedpredictability in narrative comprehension. Journal of ExperimentalPsychology: General, 138, 307–327.
Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S., & Reynolds, J. R.(2007). Event perception: A mind/brain perspective. PsychologicalBulletin, 133(2), 273–293.
Zacks, J. M., Speer, N. K., Vettel, J. M., & Jacoby, L. L. (2006a). Eventunderstanding and memory in healthy aging and dementia of theAlzheimer type. Psychology and Aging, 21(3), 466–482.
Zacks, J. M., Swallow, K. M., Speer, N. K., & Maley, C. J. (2006b). The humanbrain’s response to change in cinema. Abstracts of the PsychonomicSociety, 11, 9.
Zacks, J. M., Swallow, K. M., Vettel, J. M., & McAvoy, M. P. (2006c). Visualmovement and the neural correlates of event perception. BrainResearch, 1076(1), 150–162.
Zacks, J. M., & Tversky, B. (2001). Event structure in perception andconception. Psychological Bulletin, 127(1), 3–21.
Zacks, J. M., Tversky, B., & Iyer, G. (2001b). Perceiving, remembering, andcommunicating structure in events. Journal of ExperimentalPsychology: General, 130(1), 29–58.
Zhang, Z., & Troje, N. F. (2007). 3D periodic human motion reconstructionfrom 2D motion sequences. Neural Computation, 19(5), 1400–1421.
Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in languagecomprehension and memory. Psychological Bulletin, 123(2), 162–185.
