Two-Stream Convolutional Networks for Dynamic Texture Synthesis

Matthew Tesfaldet Marcus A. BrubakerDepartment of Electrical Engineering and Computer Science

York University, Toronto{mtesfald,mab}@eecs.yorku.ca

Konstantinos G. DerpanisDepartment of Computer Science

Ryerson University, Torontokosta@scs.ryerson.ca

Abstract

We introduce a two-stream model for dynamic texturesynthesis. Our model is based on pre-trained convolutionalnetworks (ConvNets) that target two independent tasks: (i)object recognition, and (ii) optical flow prediction. Givenan input dynamic texture, statistics of filter responses fromthe object recognition ConvNet encapsulate the per-frameappearance of the input texture, while statistics of filter re-sponses from the optical flow ConvNet model its dynamics.To generate a novel texture, a randomly initialized input se-quence is optimized to match the feature statistics from eachstream of an example texture. Inspired by recent work onimage style transfer and enabled by the two-stream model,we also apply the synthesis approach to combine the textureappearance from one texture with the dynamics of anotherto generate entirely novel dynamic textures. We show thatour approach generates novel, high quality samples thatmatch both the framewise appearance and temporal evo-lution of input texture. Finally, we quantitatively evaluateour texture synthesis approach with a thorough user study.

1. IntroductionMany common temporal visual patterns are naturally de-

scribed by the ensemble of appearance and dynamics (i.e.,temporal pattern variation) of their constituent elements.Examples of such patterns include fire, fluttering vegetation,and wavy water. Understanding and characterizing thesetemporal patterns has long been a problem of interest in hu-man perception, computer vision, and computer graphics.These patterns have been previously studied under a varietyof names, including turbulent-flow motion [17], temporaltextures [30], time-varying textures [3], dynamic textures[8], textured motion [45] and spacetime textures [7]. Here,we adopt the term “dynamic texture”. In this work, we pro-pose a factored analysis of dynamic textures in terms of ap-pearance and temporal dynamics. This factorization is thenused to enable dynamic texture synthesis which, based on

appearance target

output

dynamics target

appearance & dynamics target output

Dynamic Texture Synthesis Dynamics Style Transfer

Figure 1: Dynamic texture synthesis. (left) Given an inputdynamic texture as the target, our two-stream model is ableto synthesize a novel dynamic texture that preserves the tar-get’s appearance and dynamics characteristics. (right) Ourtwo-stream approach enables synthesis that combines thetexture appearance from one target with the dynamics fromanother, resulting in a composition of the two.

example texture inputs, generates a novel dynamic textureinstance. It also enables a novel form of style transfer wherethe target appearance and dynamics can be taken from dif-ferent sources as shown in Fig. 1.

Our model is constructed from two convolutional net-works (ConvNets), an appearance stream and a dynamicsstream, which have been pre-trained for object recognitionand optical flow prediction, respectively. Similar to previ-ous work on spatial textures [13, 19, 33], we summarize aninput dynamic texture in terms of a set of spatiotemporalstatistics of filter outputs from each stream. The appear-ance stream models the per frame appearance of the inputtexture, while the dynamics stream models its temporal dy-namics. The synthesis process consists of optimizing a ran-domly initialized noise pattern such that its spatiotemporalstatistics from each stream match those of the input tex-ture. The architecture is inspired by insights from humanperception and neuroscience. In particular, psychophysicalstudies [6] show that humans are able to perceive the struc-ture of a dynamic texture even in the absence of appearancecues, suggesting that the two streams are effectively inde-

arX

iv:1

706.

0698

2v4

[cs

.CV

] 1

2 A

pr 2

018

pendent. Similarly, the two-stream hypothesis [16] modelsthe human visual cortex in terms of two pathways, the ven-tral stream (involved with object recognition) and the dorsalstream (involved with motion processing).

In this paper, our two-stream analysis of dynamic tex-tures is applied to texture synthesis. We consider a rangeof dynamic textures and show that our approach generatesnovel, high quality samples that match both the frame-wiseappearance and temporal evolution of an input example.Further, the factorization of appearance and dynamics en-ables a novel form of style transfer, where dynamics of onetexture are combined with the appearance of a different one,cf . [14]. This can even be done using a single image asan appearance target, which allows static images to be an-imated. Finally, we validate the perceived realism of ourgenerated textures through an extensive user study.

2. Related workThere are two general approaches that have dominated

the texture synthesis literature: non-parametric samplingapproaches that synthesize a texture by sampling pixels ofa given source texture [10, 26, 37, 47], and statistical para-metric models. As our approach is an instance of a para-metric model, here we focus on these approaches.

The statistical characterization of visual textures was in-troduced in the seminal work of Julesz [23]. He conjecturedthat particular statistics of pixel intensities were sufficientto partition spatial textures into metameric (i.e., perceptu-ally indistinguishable) classes. Later work leveraged thisnotion for texture synthesis [19, 33]. In particular, inspiredby models of the early stages of visual processing, statisticsof (handcrafted) multi-scale oriented filter responses wereused to optimize an initial noise pattern to match the filterresponse statistics of an input texture. More recently, Gatyset al. [13] demonstrated impressive results by replacing thelinear filter bank with a ConvNet that, in effect, served asa proxy for the ventral visual processing stream. Texturesare modelled in terms of the correlations between filter re-sponses within several layers of the network. In subsequentwork, this texture model was used in image style transfer[14], where the style of one image was combined with theimage content of another to produce a new image. Ruder etal. [36] extended this model to video by using optical flowto enforce temporal consistency of the resulting imagery.

Variants of linear autoregressive models have been stud-ied [42, 8] that jointly model appearance and dynamics ofthe spatiotemporal pattern. More recent work has consid-ered ConvNets as a basis for modelling dynamic textures.Xie et al. [48] proposed a spatiotemporal generative modelwhere each dynamic texture is modelled as a random fielddefined by multiscale, spatiotemporal ConvNet filter re-sponses and dynamic textures are realized by sampling themodel. Unlike our current work, which assumes pretrained

fixed networks, this approach requires the ConvNet weightsto be trained using the input texture prior to synthesis.

A recent preprint [12] described preliminary results ex-tending the framework of Gatys et al. [13] to model and syn-thesize dynamic textures by computing a Gram matrix offilter activations over a small temporal window. In contrast,our two stream filtering architecture is more expressive asour dynamics stream is specifically tuned to spatiotemporaldynamics. Moreover, as will be demonstrated, the factoriza-tion in terms of appearance and dynamics enables a novelform of style transfer, where the dynamics of one patternare transferred to the appearance of another to generate anentirely new dynamic texture. To the best of our knowledge,we are the first to demonstrate this form of style transfer.

The recovery of optical flow from temporal imageryhas long been studied in computer vision. Tradition-ally, it has been addressed by handcrafted approaches e.g.,[20, 29, 35]. Recently, ConvNet approaches [9, 34, 21, 49]have been demonstrated as viable alternatives. Most closelyrelated to our approach are energy models of visual motion[2, 18, 39, 31, 7, 25] that have been motivated and studiedin a variety of contexts, including computer vision, visualneuroscience, and visual psychology. Given an input imagesequence, these models consist of an alternating sequenceof linear and non-linear operations that yield a distributedrepresentation (i.e., implicitly coded) of pixelwise opticalflow. Here, an energy model motivates the representation ofobserved dynamics which is then encoded as a ConvNet.

3. Technical approachOur proposed two-stream approach consists of an ap-

pearance stream, representing the static (texture) appear-ance of each frame, and a dynamics stream, representingtemporal variations between frames. Each stream consistsof a ConvNet whose activation statistics are used to charac-terize the dynamic texture. Synthesizing a dynamic textureis formulated as an optimization problem with the objectiveof matching the activation statistics. Our dynamic texturesynthesis approach is summarized in Fig. 2 and the individ-ual pieces are described in turn in the following sections.

3.1. Texture model: Appearance stream

The appearance stream follows the spatial texture modelintroduced by Gatys et al. [13] which we briefly reviewhere. The key idea is that feature correlations in a Con-vNet trained for object recognition capture texture appear-ance. We use the same publicly available normalized VGG-19 network [40] used by Gatys et al. [13].

To capture the appearance of an input dynamic texture,we first perform a forward pass with each frame of the im-age sequence through the ConvNet and compute the featureactivations, Alt ∈ RNl×Ml , for various levels in the net-work, where Nl and Ml denote the number of filters and

synthesized (ToutxHxWx3)

input (TxHxWx3)

appearance stream

dynamics stream

… …

… …

Ldynamic texture

A1 A1 AlAl

+

Ldynamics=

=

Lappearance=

T

T � 1

D1 D1 DlDl

LlappearanceL1

appearance

Gl GlG1 G1

} }

+… + +…

……

L1dynamics Ll

dynamics

Gl GlG1 G1

+… + +…

} } ……

Figure 2: Two-stream dynamic texture generation. Sets ofGram matrices represent a texture’s appearance and dynam-ics. Matching these statistics allows for the generation ofnovel textures as well as style transfer between textures.

the number of spatial locations of layer l at time t, respec-tively. The correlations of the filter responses in a particularlayer are averaged over the frames and encapsulated by aGram matrix, Gl ∈ RNl×Nl , whose entries are given byGl

ij = 1TNlMl

∑Tt=1

∑Ml

k=1AltikA

ltjk, where T denotes the

number of input frames and Altik denotes the activation of

feature i at location k in layer l on the target frame t. Thesynthesized texture appearance is similarly represented bya Gram matrix, Glt ∈ RNl×Nl , whose activations are givenby Glt

ij =1

NlMl

∑Ml

k=1 AltikA

ltjk, where Alt

ik denotes the acti-vation of feature i at location k in layer l on the synthesizedframe t. The appearance loss, Lappearance, is then defined asthe temporal average of the mean squared error between theGram matrix of the input texture and that of the generatedtexture computed at each frame:

Lappearance =1

LappTout

Tout∑t=1

∑l

‖Gl − Glt‖2F , (1)

where Lapp is the number of layers used to compute Grammatrices, Tout is the number of frames being generated inthe output, and ‖ · ‖F is the Frobenius norm. Consistentwith previous work [13], we compute Gram matrices on thefollowing layers: conv1 1, pool1, pool2, pool3, and pool4.

3.2. Texture model: Dynamics stream

There are three primary goals in designing our dynamicsstream. First, the activations of the network must representthe temporal variation of the input pattern. Second, the acti-vations should be largely invariant to the appearance of theimages which should be characterized by the appearancestream described above. Finally, the representation must bedifferentiable to enable synthesis. By analogy to the ap-pearance stream, an obvious choice is a ConvNet architec-

ture suited for computing optical flow (e.g., [9, 21]) whichis naturally differentiable. However, with most such mod-els it is unclear how invariant their layers are to appearance.Instead, we propose a novel network architecture which ismotivated by the spacetime-oriented energy model [7, 39].

In motion energy models, the velocity of image content(i.e., motion) is interpreted as a three-dimensional orienta-tion in the x-y-t spatiotemporal domain [2, 11, 18, 39, 46].In the frequency domain, the signal energy of a translatingpattern can be shown to lie on a plane through the originwhere the slant of the plane is defined by the velocity ofthe pattern. Thus, motion energy models attempt to identifythis orientation-plane (and hence the patterns velocity) viaa set of image filtering operations. More generally the con-stituent spacetime orientations for a spectrum of commonvisual patterns (including translation and dynamic textures)can serve as a basis for describing the temporal variation ofan image sequence [7]. This suggests that motion energymodels may form an ideal basis for our dynamics stream.

Specifically, we use the spacetime-oriented energymodel [7, 39] to motivate our network architecture whichwe briefly review here; see [7] for a more in-depth descrip-tion. Given an input video, a bank of oriented 3D filtersare applied which are sensitive to a range of spatiotemporalorientations. These filter activations are rectified (squared)and pooled over local regions to make the responses robustto the phase of the input signal, i.e., robust to the alignmentof the filter with the underlying image structure. Next, fil-ter activations consistent with the same spacetime orienta-tion are summed. These responses provide a pixelwise dis-tributed measure of which orientations (frequency domainplanes) are present in the input. However, these responsesare confounded by local image contrast that makes it dif-ficult to determine whether a high response is indicativeof the presence of a spacetime orientation or simply dueto high image contrast. To address this ambiguity, an L1

normalization is applied across orientation responses whichresults in a representation that is robust to local appearancevariations but highly selective to spacetime orientation.

Using this model as our basis, we propose the follow-ing fully convolutional network [38]. Our ConvNet in-put is a pair of temporally consecutive greyscale images.Each input pair is first normalized to have zero-mean andunit variance. This step provides a level of invariance tooverall brightness and contrast, i.e., global additive andmultiplicative signal variations. The first layer consists of32 3D spacetime convolution filters of size 11 × 11 × 2(height×width×time). Next, a squaring activation functionand 5 × 5 spatial max-pooling (with a stride of one) is ap-plied to make the responses robust to local signal phase. A1×1 convolution layer follows with 64 filters that combinesenergy measurements that are consistent with the same ori-entation. Finally, to remove local contrast dependence, an

encode

encode

encode

input (Nx2xHxWx1)

contrast norm

flow

conv(3x3)

64 filters

conv(1x1)

2 filtersReLU

decode

aEPE

target flow

channel concat

(x2)

(x4)

conv(2x11x11) 32 filters

rectifymax

stride 1

pool(5x5)

conv(1x1)

64 filters

L1

norm

encode

(x½)

(x½)decode

Figure 3: Dynamics stream ConvNet. The ConvNet is basedon a spacetime-oriented energy model [7, 39] and is trainedfor optical flow prediction. Three scales are shown for il-lustration; in practice five scales were used.

L1 divisive normalization is applied.To capture spacetime orientations beyond those capable

with the limited receptive fields used in the initial layer,we compute a five-level spatial Gaussian pyramid. Eachpyramid level is processed independently with the samespacetime-oriented energy model and then bilinearly up-sampled to the original resolution and concatenated.

Prior energy model instantiations (e.g., [2, 7, 39]) usedhandcrafted filter weights. While a similar approach couldbe followed here, we opt to learn the weights so that theyare better tuned to natural imagery. To train the networkweights, we add additional decoding layers that take theconcatenated distributed representation and apply a 3 × 3convolution (with 64 filters), ReLU activation, and a 1 × 1convolution (with 2 filters) that yields a two channel outputencoding the optical flow directly. The proposed architec-ture is illustrated in Fig. 3.

For training, we use the standard average endpoint er-ror (aEPE) flow metric (i.e., L2 norm) between the pre-dicted flow and the ground truth flow as the loss. Since nolarge-scale flow dataset exists that captures natural imagerywith groundtruth flow, we take an unlabeled video datasetand apply an existing flow estimator [35] to estimate opti-cal flow for training, cf . [43]. For training data, we usedvideos from the UCF101 dataset [41] with geometric andphotometric data augmentations similar to those used byFlowNet [9], and optimized the aEPE loss using Adam [24].Inspection of the learned filters in the initial layer showedevidence of spacetime-oriented filters, consistent with thehandcrafted filters used in previous work [7].

Similar to the appearance stream, filter response cor-relations in a particular layer of the dynamics stream areaveraged over the number of image frame pairs and en-capsulated by a Gram matrix, Gl ∈ RNl×Nl , whose en-tries are given by Gl

ij =1

(T−1)NlMl

∑T−1t=1

∑Ml

k=1DltikD

ltjk,

where Dltik denotes the activation of feature i at location k

in layer l on the target frames t and t + 1. The dynam-ics of the synthesized texture is represented by a Gram ma-trix of filter response correlations computed separately foreach pair of frames, Glt ∈ RNl×Nl , with entries Glt

ij =1

NlMl

∑Ml

k=1 DltikD

ltjk, where Dlt

ik denotes the activation offeature i at location k in layer l on the synthesized frames tand t+1. The dynamics loss, Ldynamics, is defined as the av-erage of the mean squared error between the Gram matricesof the input texture and those of the generated texture:

Ldynamics =1

Ldyn(Tout − 1)

Tout−1∑t=1

∑l

‖Gl − Glt‖2F , (2)

where Ldyn is the number of ConvNet layers being used inthe dynamics stream.

Here we propose to use the output of the concatenationlayer, where the multiscale distributed representation of ori-entations is stored, as the layer to compute the Gram ma-trix. While it is tempting to use the predicted flow out-put from the network, this generally yields poor results asshown in our evaluation. Due to the complex, temporal vari-ation present in dynamic textures, they contain a variety oflocal spacetime orientations rather than a single dominantorientation. As a result, the flow estimates will tend to bean average of the underlying orientation measurements andconsequently not descriptive. A comparison between thetexture synthesis results using the concatenation layer andthe predicted flow output is provided in Sec. 4.

3.3. Texture generation

The overall dynamic texture loss consists of the combi-nation of the appearance loss, Eq. (1), and the dynamicsloss, Eq. (2):

Ldynamic texture = αLappearance + βLdynamics, (3)

where α and β are the weighting factors for the appearanceand dynamics content, respectively. Dynamic textures areimplicitly defined as the (local) minima of this loss. Tex-tures are generated by optimizing Eq. (3) with respect tothe spacetime volume, i.e., the pixels of the video. Vari-ations in the resulting texture are found by initializing theoptimization process using IID Gaussian noise. Consistentwith previous work [13], we use L-BFGS [28] optimization.

Naive application of the outlined approach will consumeincreasing amounts of memory as the temporal extent of thedynamic texture grows; this makes it impractical to gener-ate longer sequences. Instead, long sequences can be in-crementally generated by separating the sequence into sub-sequences and optimizing them sequentially. This is real-ized by initializing the first frame of a subsequence as thelast frame from the previous subsequence and keeping itfixed throughout the optimization. The remaining frames

of the subsequence are initialized randomly and optimizedas above. This ensures temporal consistency across synthe-sized subsequences and can be viewed as a form of coordi-nate descent for the full sequence objective. The flexibilityof this framework allows other texture generation problemsto be handled simply by altering the initialization of framesand controlling which frames or frame regions are updated.

4. Experimental resultsThe goal of (dynamic) texture synthesis is to gener-

ate samples that are indistinguishable from the real inputtarget texture by a human observer. In this section, wepresent a variety of synthesis results including a user studyto quantitatively evaluate the realism of our results. Giventheir temporal nature, our results are best viewed as videos.Our two-stream architecture was implemented using Ten-sorFlow [1]. Results were generated using an NVIDIA Ti-tan X (Pascal) GPU and synthesis times ranged betweenone to three hours to generate 12 frames with an imageresolution of 256 × 256. For our full synthesis resultsand source code, please refer to the supplemental materialon the project website: ryersonvisionlab.github.io/two-stream-projpage.

4.1. Dynamic texture synthesis

We applied our dynamic texture synthesis process to awide range of textures which were selected from the Dyn-Tex [32] database and others we collected in the wild. In-cluded in our supplemental material are synthesized resultsof nearly 60 different textures that encapsulate a range ofphenomena, such as flowing water, waves, clouds, fire, rip-pling flags, waving plants, and schools of fish. Some sam-ple frames are shown in Fig. 4 but we encourage readers toview the videos to fully appreciate the results. In addition,we performed a comparison with [12] and [48]. Generally,we found our results to be qualitatively comparable or betterthan these methods. See the supplemental for more detailson the comparisons with these methods.

We also generated dynamic textures incrementally, asdescribed in Sec. 3.3. The resulting textures were perceptu-ally indistinguishable from those generated with the batchprocess. Another extension that we explored were textureswith no discernible temporal seam between the last and firstframes. Played as a loop, these textures appear to be tempo-rally endless. This was achieved by assuming that the firstframe follows the final frame and adding an additional lossfor the dynamics stream evaluated on that pair of frames.

Example failure modes of our method are presentedin Fig. 6. In general, we find that most failures resultfrom inputs that violate the underlying assumption of adynamic texture, i.e., the appearance and/or dynamics arenot spatiotemporally homogeneous. In the case of theescalator example, the long edge structures in the ap-

pearance are not spatially homogeneous, and the dynam-ics vary due to perspective effects that change the motionfrom downward to outward. The resulting synthesized tex-ture captures an overall downward motion but lacks the per-spective effects and is unable to consistently reproduce thelong edge structures. This is consistent with previous ob-servations on static texture synthesis [13] and suggests it isa limitation of the appearance stream.

Another example is the flag sequence where the rip-pling dynamics are relatively homogeneous across the pat-tern but the appearance varies spatially. As expected, thegenerated texture does not faithfully reproduce the appear-ance; however, it does exhibit plausible rippling dynamics.In the supplemental material, we include an additional fail-ure case, cranberries, which consists of a swirling pat-tern. Our model faithfully reproduces the appearance but isunable to capture the spatially varying dynamics. Interest-ingly, it still produces a result which is statistically indistin-guishable from real in our user study discussed below.

Appearance vs. dynamics streams We sought to verifythat the appearance and dynamics streams were capturingcomplementary information. To validate that the texturegeneration of multiple frames would not induce dynamicsconsistent with the input, we generated frames starting fromrandomly generated noise but only using the appearancestatistics and corresponding loss, i.e., Eq. 1. As expected,this produced frames that were valid textures but with nocoherent dynamics present. Results for a sequence contain-ing a school of fish are shown in Fig. 5; to examine thedynamics, see fish in the supplemental material.

Similarly, to validate that the dynamics stream did not in-advertently include appearance information, we generatedvideos using the dynamics loss only, i.e., Eq. 2. The re-sulting frames had no visible appearance and had an ex-tremely low dynamic range, i.e., the standard deviation ofpixel intensities was 10 for values in [0, 255]. This indi-cates a general invariance to appearance and suggests thatour two-stream dynamic texture representation has factoredappearance and dynamics, as desired.

4.2. User study

Quantitative evaluation for texture synthesis is a partic-ularly challenging task as there is no single correct outputwhen synthesizing new samples of a texture. Like in otherimage generation tasks (e.g., rendering), human perceptionis ultimately the most important measure. Thus, we per-formed a user study to evaluate the perceived realism of oursynthesized textures.

Similar to previous image synthesis work (e.g., [5]), weconducted a perceptual experiment with human observersto quantitatively evaluate our synthesis results. We em-ployed a forced-choice evaluation on Amazon Mechanical

fireplace 1

(original)

fireplace 1

(synthesized)

lava

(original)

lava

(synthesized)

smoke 1

(original)

smoke 1

(synthesized)

underwater

vegetation 1

(original)

underwater

vegetation 1

(synthesized)

water 3

(original)

water 3

(synthesized)

Figure 4: Dynamic texture synthesis success examples. Names correspond to files in the supplemental material.

Turk (AMT) with 200 different users. Each user performed59 pairwise comparisons between a synthesized dynamictexture and its target. Users were asked to choose whichappeared more realistic after viewing the textures for an ex-posure time sampled randomly from discrete intervals be-tween 0.3 and 4.8 seconds. Measures were taken to controlthe experimental conditions and minimize the possibility oflow quality data. See the supplemental material for furtherexperimental details of our user study.

For comparison, we constructed a baseline by using theflow decode layer in the dynamics loss of Eq. 2. This corre-sponds with attempting to mimic the optical flow statisticsof the texture directly. Textures were synthesized with thismodel and the user study was repeated with an additional200 users. To differentiate between the models, we label

“Flow decode layer” and “Concat layer” in the figures todescribe our baseline and final model, respectively.

The results of this study are summarized in Fig. 7 whichshows user accuracy in differentiating real versus generatedtextures as a function of time for both methods. Over-all, users are able to correctly identify the real texture66.1% ± 2.5% of the time for brief exposures of 0.3 sec-onds. This rises to 79.6%±1.1% with exposures of 1.2 sec-onds and higher. Note that “perfect” synthesis results wouldhave an accuracy of 50%, indicating that users were unableto differentiate between the real and generated textures andhigher accuracy indicating less convincing textures.

The results clearly show that the use of the concatenationlayer activations is far more effective than the flow decodelayer. This is not surprising as optical flow alone is known

target(fish)

appearanceonly

bothstreams

Figure 5: Dynamic texture synthesis versus texture synthe-sis. (top row) Target texture. (middle) Texture synthesiswithout dynamics constraints shows consistent per-frameappearance but no temporal coherence. (bottom) Includingboth streams induces consistent appearance and dynamics.

escalator

(original)

escalator

(synthesized)

flag

(original)

flag

(synthesized)

Figure 6: Dynamic texture synthesis failure examples. Inthese cases, the failures are attributed to either the appear-ance or the dynamics not being homogeneous.

to be unreliable on many textures, particularly those withtransparency or chaotic motion (e.g., water, smoke, flames,etc.). Also evident in these results is the time-dependantnature of perception for textures from both models. Users’ability to identify the generated texture improved as expo-sure times increased to 1.2 seconds and remained relativelyflat for longer exposures.

To better understand the performance of our approach,we grouped and analyzed the results in terms of appear-ance and dynamics characteristics. For appearance we usedthe taxonomy presented in [27] and grouped textures aseither regular/near-regular (e.g., periodic tiling and brickwall), irregular (e.g., a field of flowers), or stochastic/near-stochastic (e.g., tv static or water). For dynamics wegrouped textures as either spatially-consistent (e.g., closeupof rippling sea water) or spatially-inconsistent (e.g., ripplingsea water juxtaposed with translating clouds in the sky). Re-

300

400

600

1200

2400

3600

4800

Exposure time (ms)

0.5

0.6

0.7

0.8

0.9

1.0

Accu

racy

Flow decode layerConcat layer

Figure 7: Time-limited pairwise comparisons across all tex-tures with 95% statistical confidence intervals.

sults based on these groupings can be seen in Fig. 8.

A full breakdown of the user study results by texture andgrouping can be found in the supplemental material. Herewe discuss some of the overall trends. Based on appear-ance it is clear that textures with large-scale spatial consis-tencies (regular, near-regular, and irregular textures) tend toperform poorly. Examples being flag and fountain 2with user accuracies of 98.9% ± 1.6% and 90.8% ± 4.3%averaged across all exposures, respectively. This is not un-expected and is a fundamental limitation of the local na-ture of the Gram matrix representation used in the appear-ance stream which was observed in static texture synthe-sis [13]. In contrast, stochastic and near-stochastic texturesperformed significantly better as their smaller-scale localvariations are well captured by the appearance stream, forinstance water 1 and lava which had average accuraciesof 53.8%± 7.4% and 55.6%± 7.4%, respectively, makingthem both statistically indistinguishable from real.

In terms of dynamics, we find that textures withspatially-consistent dynamics (e.g., tv static,water *, and calm water *) perform significantlybetter than those with spatially-inconsistent dynamics (e.g.,candle flame, fountain 2, and snake *), wherethe dynamics drastically differ across spatial locations.For example, tv static and calm water 6 haveaverage accuracies of 48.6% ± 7.4% and 63.2% ± 7.2%,respectively, while candle flame and snake 5 haveaverage accuracies of 92.4% ± 4% and 92.1% ± 4%,respectively. Overall, our model is capable of reproducinga full spectrum of spatially-consistent dynamics. However,as the appearance shifts from containing small-scale spatialconsistencies to containing large-scale consistencies,performance degrades. This was evident in the user studywhere the best-performing textures typically consisted ofa stochastic or near-stochastic appearance with spatially-consistent dynamics. In contrast the worst-performingtextures consisted of regular, near-regular, or irregularappearance with spatially-inconsistent dynamics.

300

400

600

1200

2400

3600

4800

Exposure time (ms)

0.5

0.6

0.7

0.8

0.9

1.0Ac

cura

cy

Regular & Near-regularIrregularStochastic & Near-stochastic

300

400

600

1200

2400

3600

4800

Exposure time (ms)

0.5

0.6

0.7

0.8

0.9

1.0

Accu

racy

Spatially-consistentSpatially-inconsistent

Figure 8: Time-limited pairwise comparisons across all tex-tures, grouped by appearance (top) and dynamics (bottom).Shown with 95% statistical confidence intervals.

4.3. Dynamics style transfer

The underlying assumption of our model is that appear-ance and dynamics of texture can be factorized. As such, itshould allow for the transfer of the dynamics of one textureonto the appearance of another. This has been explored pre-viously for artistic style transfer [4, 15] with static imagery.We accomplish this with our model by performing the sameoptimization as above, but with the target Gram matrices forappearance and dynamics computed from different textures.

A dynamics style transfer result is shown in Fig. 9 (top),using two real videos. Additional examples are availablein the supplemental material. We note that when perform-ing dynamics style transfer it is important that the appear-ance structure be similar in scale and semantics, otherwise,the generated dynamic textures will look unnatural. For in-stance, transferring the dynamics of a flame onto a waterscene will generally produce implausible results.

We can also apply the dynamics of a texture to a staticinput image, as the target Gram matrices for the appearanceloss can be computed on just a single frame. This allows usto effectively animate regions of a static image. The resultof this process can be striking and is visualized in Fig. 9(bottom), where the appearance is taken from a painting andthe dynamics from a real world video.

5. Discussion and summaryIn this paper, we presented a novel, two-stream model of

dynamic textures using ConvNets to represent the appear-

appearancetarget synthesized output

Figure 9: Dynamics style transfer. (top row) Appearance ofstill water was used with the dynamics of a different waterdynamic texture (water 4). (bottom row) The appearanceof a painting of fire was used with the dynamics of a realfire (fireplace 1). Animated results and additional ex-amples are available in the supplemental material.

ance and dynamics. We applied this model to a variety ofdynamic texture synthesis tasks and showed that, so longas the input textures are generally true dynamic textures,i.e., have spatially invariant statistics and spatiotemporallyinvariant dynamics, the resulting synthesized textures arecompelling. This was validated both qualitatively and quan-titatively through a large user study. Further, we showedthat the two-stream model enabled dynamics style transfer,where the appearance and dynamics information from dif-ferent sources can be combined to generate a novel texture.

We have explored this model thoroughly and found a fewlimitations which we leave as directions for future work.First, much like has been reported in recent image styletransfer work [14], we have found that high frequency noiseand chromatic aberrations are a problem in generation. An-other issue that arises is the model fails to capture textureswith spatially-variant appearance, (e.g., flag in Fig. 6) andspatially-inconsistent dynamics (e.g., escalator in Fig.6). By collapsing the local statistics into a Gram matrix,the spatial and temporal organization is lost. Simple post-processing methods may alleviate some of these issues butwe believe that they also point to a need for a better rep-resentation. Beyond addressing these limitations, a naturalnext step would be to extend the idea of a factorized rep-resentation into feed-forward generative networks that havefound success in static image synthesis, e.g., [22, 44].

Acknowledgements MT is supported by a Natural Sci-ences and Engineering Research Council of Canada(NSERC) Canadian Graduate Scholarship. KGD and MABare supported by NSERC Discovery Grants. This researchwas undertaken as part of the Vision: Science to Applica-tions program, thanks in part to funding from the CanadaFirst Research Excellence Fund.

References[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,

C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghe-mawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia,R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane,R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster,J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,V. Vanhoucke, V. Vasudevan, F. Viegas, O. Vinyals, P. War-den, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensor-Flow: Large-scale machine learning on heterogeneous sys-tems, 2015. Software available from tensorflow.org. 5

[2] E. H. Adelson and J. R. Bergen. Spatiotemporal energy mod-els for the perception of motion. JOSA–A, 2(2):284–299,1985. 2, 3, 4

[3] Z. Bar-Joseph, R. El-Yaniv, D. Lischinski, and M. Werman.Texture mixing and texture movie synthesis using statisticallearning. T-VCG, 7(2):120–135, 2001. 1

[4] A. J. Champandard. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv:1603.01768, 2016. 8

[5] Q. Chen and V. Koltun. Photographic image synthesis withcascaded refinement networks. In ICCV, 2017. 5

[6] J. E. Cutting. Blowing in the wind: Perceiving structure intrees and bushes. Cognition, 12(1):25 – 44, 1982. 1

[7] K. G. Derpanis and R. P. Wildes. Spacetime texture represen-tation and recognition based on a spatiotemporal orientationanalysis. PAMI, 34(6):1193–1205, 2012. 1, 2, 3, 4

[8] G. Doretto, A. Chiuso, Y. N. Wu, and S. Soatto. Dynamictextures. IJCV, 51(2):91–109, 2003. 1, 2

[9] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas,V. Golkov, P. van der Smagt, D. Cremers, and T. Brox.FlowNet: Learning optical flow with convolutional net-works. In ICCV, pages 2758–2766, 2015. 2, 3, 4

[10] A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In ICCV, pages 1033–1038, 1999. 2

[11] M. Fahle and T. Poggio. Visual hyperacuity: Spatiotemporalinterpolation in human vision. Proceedings of the Royal So-ciety of London B: Biological Sciences, 213(1193):451–477,1981. 3

[12] C. M. Funke, L. A. Gatys, A. S. Ecker, and M. Bethge. Syn-thesising dynamic textures using convolutional neural net-works. arXiv:1702.07006, 2017. 2, 5, 10, 11

[13] L. A. Gatys, A. S. Ecker, and M. Bethge. Texture synthesisusing convolutional neural networks. In NIPS, pages 262–270, 2015. 1, 2, 3, 4, 5, 7

[14] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transferusing convolutional neural networks. In CVPR, pages 2414–2423, 2016. 2, 8

[15] L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, andE. Shechtman. Controlling perceptual factors in neural styletransfer. In CVPR, 2017. 8

[16] M. A. Goodale and A. D. Milner. Separate visual path-ways for perception and action. Trends in Neurosciences,15(1):20–25, 1992. 2

[17] D. Heeger and A. Pentland. Seeing structure through chaos.In IEEE Motion Workshop: Representation and Analysis,pages 131–136, 1986. 1

[18] D. J. Heeger. Optical flow using spatiotemporal filters. IJCV,1(4):279–302, 1988. 2, 3

[19] D. J. Heeger and J. R. Bergen. Pyramid-based texture analy-sis/synthesis. In SIGGRAPH, pages 229–238, 1995. 1, 2

[20] B. K. P. Horn and B. G. Schunck. Determining optical flow.A.I., 17:185–203, 1981. 2

[21] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, andT. Brox. FlowNet 2.0: Evolution of optical flow estimationwith deep networks. In CVPR, 2017. 2, 3

[22] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses forreal-time style transfer and super-resolution. In ECCV, pages694–711, 2016. 8

[23] B. Julesz. Visual pattern discrimination. IRE Trans. Infor-mation Theory, 8(2):84–92, 1962. 2

[24] D. P. Kingma and J. Ba. Adam: A Method for StochasticOptimization. arXiv:1412.6980, 2014. 4

[25] K. Konda, R. Memisevic, and V. Michalski. Learning to en-code motion using spatio-temporal synchrony internationalconference on learning representation. In ICLR, 2014. 2

[26] V. Kwatra, A. Schodl, I. Essa, G. Turk, and A. Bobick.Graphcut textures: Image and video synthesis using graphcuts. In SIGGRAPH, pages 277–286, 2003. 2

[27] W.-C. Lin, J. Hays, C. Wu, Y. Liu, and V. Kwatra. Quantita-tive evaluation of near regular texture synthesis algorithms.In CVPR, volume 1, pages 427–434, 2006. 7

[28] D. C. Liu and J. Nocedal. On the limited memory methodfor large scale optimization. Mathematical Programming,45(3):503–528, 1989. 4

[29] B. D. Lucas and T. Kanade. An iterative image registra-tion technique with an application to stereo vision. In IJCAI,pages 674–679, 1981. 2

[30] R. Nelson and R. Polana. Qualitative recognition of motionusing temporal textures. CVGIP, 56(1), 1992. 1

[31] S. Nishimoto and J. L. Gallant. A three-dimensional spa-tiotemporal receptive field model explains responses of areamt neurons to naturalistic movies. Journal of Neuroscience,31(41):14551–14564, 2011. 2

[32] R. Peteri, S. Fazekas, and M. J. Huiskes. DynTex: A Com-prehensive Database of Dynamic Textures. PRL, 31(12),2010. 5

[33] J. Portilla and E. P. Simoncelli. A parametric texture modelbased on joint statistics of complex wavelet coefficients.IJCV, 40(1):49–70, 2000. 1, 2

[34] A. Ranjan and M. J. Black. Optical Flow Estimation using aSpatial Pyramid Network. In CVPR, 2017. 2

[35] J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid.EpicFlow: Edge-preserving interpolation of correspon-dences for optical flow. In CVPR, pages 1164–1172, 2015.2, 4

[36] M. Ruder, A. Dosovitskiy, and T. Brox. Artistic style transferfor videos. In GCPR, pages 26–36, 2016. 2

[37] A. Schodl, R. Szeliski, D. Salesin, and I. A. Essa. Videotextures. In SIGGRAPH, pages 489–498, 2000. 2

[38] E. Shelhamer, J. Long, and T. Darrell. Fully convolutionalnetworks for semantic segmentation. PAMI, 39(4):640–651,2017. 3

[39] E. P. Simoncelli and D. J. Heeger. A model of neuronal re-sponses in visual area MT. Vision Research, 38(5):743 – 761,1998. 2, 3, 4

[40] K. Simonyan and A. Zisserman. Very deep con-volutional networks for large-scale image recognition.arXiv:1409.1556, 2014. 2

[41] K. Soomro, A. R. Zamir, and M. Shah. UCF101: Adataset of 101 human actions classes from videos in the wild.arXiv:1212.0402, 2012. 4

[42] M. Szummer and R. W. Picard. Temporal texture modeling.In ICIP, pages 823–826, 1996. 2

[43] D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, andM. Paluri. Deep end2end voxel2voxel prediction. In CVPRWorkshops, pages 402–409, 2016. 4

[44] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. S. Lempitsky.Texture networks: Feed-forward synthesis of textures andstylized images. In ICML, pages 1349–1357, 2016. 8

[45] Y. Wang and S. C. Zhu. Modeling textured motion: Particle,wave and sketch. In ICCV, pages 213–220, 2003. 1

[46] A. B. Watson and A. J. Ahumada. A look at motion in thefrequency domain. In Motion workshop: Perception and rep-resentation, pages 1–10, 1983. 3

[47] L. Wei and M. Levoy. Fast texture synthesis using tree-structured vector quantization. In SIGGRAPH, pages 479–488, 2000. 2

[48] J. Xie, S.-C. Zhu, and Y. N. Wu. Synthesizing dynamicpatterns by spatial-temporal generative convnet. In CVPR,2017. 2, 5, 11

[49] J. J. Yu, A. W. Harley, and K. G. Derpanis. Back to Ba-sics: Unsupervised Learning of Optical Flow via BrightnessConstancy and Motion Smoothness. In ECCVW, 2016. 2

A. Experimental procedureHere we provide further experimental details of our user study

using Amazon Mechanical Turk (AMT). Experimental trials weregrouped into batches of Human Intelligence Tasks (HITs) for usersto complete. Each HIT consisted of 59 pairwise comparisons be-tween a synthesized dynamic texture and its target. Users wereasked to choose which texture appeared more realistic after view-ing each texture independently for an exposure time (in seconds)sampled randomly from the set {0.3, 0.4, 0.6, 1.2, 2.4, 3.6, 4.8}.Note that 12 frames of the dynamic texture corresponds to 1.2 sec-onds, i.e., 10 frames per second. Before viewing a dynamic tex-ture, a centred dot is flashed twice to indicate to the user whereto look (left or right). To prepare users for the task, the first threecomparisons were used for warm-up, exposing them to the short-est (0.3s), median (1.2s), and longest (4.8s) durations. To preventspamming and bias, we constrained the experiment as follows:users could make a choice only after both dynamic textures wereshown; the next texture comparison could only be made after adecision was made for the current comparison; a choice could notbe changed after the next pair of dynamic textures were shown;and users were each restricted to a single HIT. Obvious unrealisticdynamic textures were synthesized by terminating synthesis early(100 iterations) and were used as sentinel tests. Three of the 59pairwise comparisons were sentinels and results from users whichgave incorrect answers on any of the sentinel comparisons were

not used. The left-right order of textures within a pair, displayorder within a pair, and order of pairs within a HIT, were random-ized. An example of a HIT is shown in a video included with thesupplemental on the project page: HIT example.mp4.

Users were paid $2 USD per HIT, and were required to haveat least a 98% HIT approval rating, greater than or equal to 5000HITs approved, and to be residing in the US. We collected resultsfrom 200 unique users to evaluate our final model and another 200to evaluate our baseline model.

B. Qualitative resultsWe provide videos showcasing the qualitative results of our

two-stream model, including the experiments mentioned in themain manuscript, on our project page: ryersonvisionlab.github.io/two-stream-projpage. The videos are inMP4 format (H.264 codec) and are best viewed in a loop. Theyare enclosed in the following folders:

• target textures: This folder contains the 59 dynamictextures used as targets for synthesis.

• dynamic texture synthesis: This folder containssynthesized dynamic textures where the appearance and dy-namics targets are the same.

• using concatenation layer: This folder containssynthesized dynamic textures where the concatenation layerwas used for computing the Gramian on the dynamicsstream. These are the results from our final model.

• using flow decode layer: This folder contains syn-thesized dynamic textures where the predicted flow outputis used for computing the Gramian on the dynamics stream.These are the results from our baseline.

• full synthesis: This folder contains regularly-synthesized dynamic textures, i.e., not incrementally-generated, nor temporally-endless, etc.

• appearance stream only: This folder contains dy-namic textures synthesized using only the appearance streamof our two-stream model. The dynamics stream is not used.

• incrementally generated: This folder contains dy-namic textures synthesized using the incremental processoutlined in Section 3.3 in the main manuscript.

• temporally endless: This folder contains a synthe-sized dynamic texture (smoke plume 1) where there is nodiscernible temporal seam between the last and first frames.Played as a loop, it appears to be temporally endless, thus, itis presented in animated GIF format.

• dynamics style transfer: This folder contains syn-thesized dynamic textures where the appearance and dynam-ics targets are different. Also included are videos where thesynthesized dynamic texture is “pasted” back onto the origi-nal image it was cropped from, showing a proof-of-conceptof dynamics style transfer as an artistic tool.

• comparisons/funke: This folder contains four dy-namic texture synthesis comparisons between our model anda recent (unpublished) approach [12]. The dynamic textures

chosen are those reported by Funke et al. [12] which ex-hibit spatiotemporal homogeneity. For ease of comparison,we have concatenated the results from both models with theircorresponding targets.

• comparisons/xie and funke: This folder containsnine dynamic texture synthesis comparisons between ourmodel, Funke et al.’s [12], and Xie et al.’s [48]. The dynamictextures chosen cover the full range of our appearance anddynamics groupings. For ease of comparison, we have con-catenated the results from all models with their correspond-ing targets.

C. Full user study resultsFigures 10a and 10b show histograms of the average user ac-

curacy on each texture, averaged over a range of exposure times.The histogram bars are ordered from lowest to highest accuracy,based on the results when using our final model.

Tables 1 and 2 show the average user accuracy on each texturewhen using our final model. The results are averaged over expo-sure times. Similarly, Tables 3 and 4 show the results when usingour baseline.

Tables 5 and 6 show the average user accuracy on texture ap-pearance groups when using our final model. The results are av-eraged over exposure times. Similarly, Tables 7 and 8 show theresults when using our baseline.

Tables 9 and 10 show the average user accuracy on texture dy-namics groups when using our final model. The results are aver-aged over exposure times. Similarly, Tables 11 and 12 show theresults when using our baseline.

Tables 13 and 14 show the average user accuracy over all tex-tures when using our final model. The results are averaged overexposure times. Similarly, Tables 15 and 16 show the results whenusing our baseline.

D. Qualitative comparisonsWe qualitatively compare our results to those of Funke

et al. [12] and Xie et al. [48]. Note that Funke et al.[12] provided results on only five textures and of those onlyfour are dynamic textures in the sense that their appear-ance and dynamics are spatiotemporally coherent. Their re-sults on these sequences (cranberries, flames, leaves,and water 5) are included in the folder funke underdynamic texture synthesis/comparisons. Our re-sults are included as well.

We also compare our results to [12, 48] on nine dynamic tex-tures chosen to cover the full range of our dynamics and appear-ance groupings. We use their publicly available code and followthe parameters used in their experiments. For Funke et al.’s model[12], the parameters used are ∆t = 4 and T = 12 (recall thattarget dynamic textures consist of 12 frames). For the spatiotem-poral and temporal models from Xie et al. [48], the parametersused are T = 1200 and M = 3. A comparison between ourresults, Funke et al.’s [12], and Xie et al’s [48] on the nine dy-namic textures are included in the folder xie and funke un-der dynamic texture synthesis/comparisons. Notefor Xie et al. [48], we compare with their spatiotemporal model

(labeled “Xie et al. (ST)”) designed for dynamic textures with bothspatial and temporal homogeneity, and their temporal model (la-beled “Xie et al. (FC)”) designed for dynamic textures with onlytemporal homogeneity.

Overall, we demonstrate that our results appear qualitativelybetter, showing more temporal coherence and similarity in dy-namics and fewer artifacts, e.g., blur and flicker. This may be anatural consequence of their limited representation of dynamics.Although the spatiotemporal model of Xie et al. [48] is able tosynthesize dynamic textures that lack spatial homogeneity (e.g.,bamboo and escalator), we note that their method can notsynthesize novel dynamic textures, i.e., it appears to faithfully re-produce the target texture, reducing the applicability of their ap-proach.

As a consequence of jointly modelling appearance and dynam-ics, the methods of [12, 48] are not capable of the novel form ofstyle transfer we demonstrated. This was enabled by the factoredrepresentation of dynamics and appearance. Furthermore, the spa-tiotemporal extent of the output sequence generated by Xie et al.’s[48] method is limited to being equal to the input. The proposedapproach does not share this limitation.

lava

waterfa

ll_2

cranberr

ieswate

r_1tv_

static

fur

flush

ing_wate

r water_4

candy_

2an

ts

calm

_wate

r_2

underwate

r_veg

etatio

n_1se

a_2 wate

r_2 calm

_wate

r smoke

_2 waterfa

llinkfis

h fountain_1 water_5

plants

calm

_wate

r_5

calm

_wate

r_3 smoke

_1

calm

_wate

r_6bird

ssky

_clouds_

1

calm

_wate

r_4wate

r_3 shiny_

circle

s

boiling_w

ater_2

grass_

2 snak

e_4

sea_

1 esca

lator

coral sn

ake_

2 grass_

1

smoke

_plume_

1 snak

e_3 sn

ake_

1 bamboo

shower_

water_1

candy_

1

boiling_w

ater_1

flames sm

oke_3

candle_

flame

fountain_2

grass_

3 snak

e_5

sky_c

louds_2

fireplac

e_1

flag

flag_2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy

Flow

dec

ode

laye

rCo

ncat

laye

r

(a)S

hort

expo

sure

times

(300

-600

ms)

.

tv_sta

tic waterfa

ll_2

cranberr

ieswate

r_1

calm

_wate

r_6lav

afis

h smoke

_2sky

_clouds_

1

fur smoke

_1 water_5an

ts candy_

2

calm

_wate

r_2

calm

_wate

r_5wate

r_2se

a_2

birds

calm

_wate

r

calm

_wate

r_3 shiny_

circle

s water_3

waterfa

llfountai

n_1

underwate

r_veg

etatio

n_1

calm

_wate

r_4

ink grass_

1 water_4co

ralplan

ts

boiling_w

ater_2

smoke

_plume_

1fla

mes grass_

2 snak

e_2

boiling_w

ater_1 es

calat

orse

a_1 sn

ake_

4

flush

ing_wate

r

shower_

water_1 sm

oke_3 bamboo

sky_c

louds_2 sn

ake_

3 candy_

1 grass_

3 snak

e_1 sn

ake_

5fountai

n_2ca

ndle_fla

mefire

place_

1fla

g_2fla

g

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy

Flow

dec

ode

laye

rCo

ncat

laye

r

(b)L

ong

expo

sure

times

(120

0-48

00m

s).

Figu

re10

:Per

-tex

ture

accu

raci

esav

erag

edov

erex

posu

retim

es.E

ach

text

ure

accu

racy

incl

udes

am

argi

nof

erro

rwith

a95%

stat

istic

alco

nfide

nce.

Dynamic texture 300 ms.ants 0.625±0.194bamboo 0.769±0.162birds 0.609±0.199boiling water 1 0.806±0.139boiling water 2 0.533±0.252calm water 0.607±0.181calm water 2 0.44±0.195calm water 3 0.813±0.135calm water 4 0.727±0.186calm water 5 0.609±0.199calm water 6 0.6±0.248candle flame 0.806±0.139candy 1 0.81±0.168candy 2 0.5±0.219coral 0.591±0.205cranberries 0.48±0.196escalator 0.792±0.162fireplace 1 0.909±0.12fish 0.571±0.212flag 1.0±0.0flag 2 0.964±0.069flames 0.72±0.176flushing water 0.5±0.209fountain 1 0.435±0.203fountain 2 0.929±0.095fur 0.452±0.175grass 1 0.813±0.135grass 2 0.632±0.217grass 3 0.8±0.175ink 0.476±0.214lava 0.458±0.199plants 0.632±0.217sea 1 0.6±0.192sea 2 0.542±0.199shiny circles 0.517±0.182shower water 1 0.767±0.151sky clouds 1 0.667±0.202sky clouds 2 0.792±0.162smoke 1 0.538±0.192smoke 2 0.478±0.204smoke 3 0.769±0.162smoke plume 1 0.724±0.163snake 1 0.862±0.126snake 2 0.72±0.176snake 3 0.643±0.177snake 4 0.643±0.177snake 5 0.826±0.155tv static 0.538±0.192underwater vegetation 1 0.656±0.165water 1 0.556±0.23water 4 0.375±0.237water 2 0.632±0.217water 3 0.545±0.208water 5 0.688±0.161waterfall 0.571±0.183waterfall 2 0.444±0.187

400 ms.0.333±0.1610.786±0.2150.786±0.1520.88±0.1270.842±0.1640.571±0.2120.621±0.1770.5±0.2450.654±0.1830.773±0.1750.773±0.1750.75±0.2120.839±0.1290.429±0.2120.81±0.1680.318±0.1950.733±0.1580.952±0.0910.65±0.2091.0±0.01.0±0.00.909±0.120.565±0.2030.688±0.2270.826±0.1550.538±0.1920.778±0.1920.667±0.2020.903±0.1040.714±0.1670.346±0.1830.667±0.2020.769±0.1620.625±0.1680.741±0.1650.903±0.1040.737±0.1980.938±0.1190.731±0.170.727±0.1860.833±0.1490.783±0.1690.704±0.1720.708±0.1820.773±0.1750.815±0.1470.947±0.10.63±0.1820.5±0.2310.32±0.1830.586±0.1790.64±0.1880.741±0.1650.667±0.2180.586±0.1790.364±0.201

600 ms.0.714±0.1930.842±0.1640.615±0.1870.846±0.1960.7±0.1640.615±0.1870.622±0.1560.667±0.1690.65±0.2090.591±0.2050.643±0.1771.0±0.00.788±0.1390.727±0.1860.826±0.1550.593±0.1850.696±0.1880.897±0.1110.656±0.1650.964±0.0691.0±0.00.913±0.1150.552±0.1810.808±0.1510.815±0.1470.621±0.1770.667±0.2020.767±0.1510.95±0.0960.679±0.1730.556±0.230.652±0.1950.826±0.1550.581±0.1740.8±0.1750.75±0.160.613±0.1710.97±0.0580.741±0.1650.6±0.2150.938±0.1190.81±0.1680.826±0.1550.813±0.1910.917±0.1110.714±0.1930.889±0.1030.423±0.190.579±0.2220.667±0.1690.652±0.1950.52±0.1960.75±0.1730.586±0.1790.688±0.2270.583±0.197

1200 ms.0.536±0.1850.906±0.1010.542±0.1990.714±0.1930.87±0.1380.636±0.1640.7±0.2010.7±0.2010.767±0.1510.609±0.1990.5±0.20.909±0.120.9±0.1310.636±0.1640.815±0.1470.64±0.1880.967±0.0640.917±0.1110.652±0.1950.968±0.0620.923±0.1020.889±0.1190.871±0.1180.833±0.1491.0±0.00.75±0.150.792±0.1620.88±0.1270.958±0.080.724±0.1630.733±0.1580.767±0.1510.955±0.0870.75±0.1730.609±0.1991.0±0.00.72±0.1760.957±0.0830.471±0.2370.72±0.1760.821±0.1420.963±0.0710.88±0.1270.958±0.080.87±0.1381.0±0.00.875±0.1320.615±0.1870.821±0.1420.727±0.1860.826±0.1550.739±0.1790.833±0.1490.759±0.1560.792±0.1620.75±0.16

2400 ms.0.636±0.2010.95±0.0960.867±0.1220.97±0.0580.731±0.170.75±0.190.652±0.1950.824±0.1810.875±0.1320.708±0.1820.519±0.1881.0±0.00.938±0.1190.652±0.1950.773±0.1750.548±0.1750.933±0.1261.0±0.00.696±0.1881.0±0.01.0±0.00.889±0.1190.92±0.1060.788±0.1390.905±0.1260.737±0.1980.735±0.1481.0±0.01.0±0.00.808±0.1510.593±0.1850.806±0.1390.857±0.150.75±0.190.9±0.1310.952±0.0910.652±0.1950.92±0.1060.895±0.1380.5±0.1730.931±0.0920.84±0.1440.905±0.1260.852±0.1340.913±0.1150.917±0.1110.923±0.1020.227±0.1750.813±0.1910.571±0.2120.706±0.1530.667±0.2020.771±0.1390.65±0.2090.696±0.1880.37±0.182

3600 ms.0.857±0.150.938±0.0840.682±0.1950.96±0.0770.852±0.1340.762±0.1820.773±0.1750.63±0.1820.848±0.1220.724±0.1630.765±0.2021.0±0.00.963±0.0710.724±0.1630.885±0.1230.519±0.1880.926±0.0990.962±0.0740.692±0.1771.0±0.01.0±0.00.875±0.1320.917±0.1110.667±0.1890.967±0.0640.526±0.2250.895±0.1380.88±0.1270.92±0.1060.783±0.1690.522±0.2040.857±0.150.964±0.0690.533±0.2520.767±0.1510.87±0.1380.571±0.2590.889±0.1190.76±0.1670.724±0.1630.968±0.0620.778±0.1571.0±0.00.9±0.1071.0±0.00.889±0.1191.0±0.00.619±0.2080.733±0.1580.583±0.1970.818±0.1610.724±0.1630.652±0.1950.652±0.1950.731±0.170.632±0.217

4800 ms.0.704±0.1720.926±0.0990.778±0.1920.963±0.0711.0±0.00.762±0.1820.706±0.2170.781±0.1430.682±0.1950.786±0.1520.658±0.1510.968±0.0620.952±0.0910.741±0.1650.828±0.1370.524±0.2140.815±0.1471.0±0.00.5±0.1791.0±0.00.966±0.0660.833±0.1331.0±0.00.808±0.1510.933±0.0890.667±0.2180.826±0.1550.813±0.1350.889±0.1190.87±0.1380.652±0.1950.96±0.0770.88±0.1270.808±0.1510.652±0.1950.889±0.1450.714±0.150.962±0.0740.588±0.1650.63±0.1821.0±0.00.87±0.1381.0±0.00.88±0.1270.964±0.0690.852±0.1341.0±0.00.333±0.1780.821±0.1420.394±0.1670.917±0.1110.7±0.1640.682±0.1950.667±0.1890.833±0.1330.452±0.175

Table 1: Per-texture accuracies averaged over exposure times, using the concatenation layer. Each texture accuracy includesa margin of error with a 95% statistical confidence.

Dynamic texture Short (300-600 ms.)ants 0.526±0.111bamboo 0.797±0.103birds 0.675±0.105boiling water 1 0.841±0.086boiling water 2 0.703±0.112calm water 0.6±0.111calm water 2 0.571±0.102calm water 3 0.692±0.102calm water 4 0.676±0.111calm water 5 0.657±0.114calm water 6 0.677±0.114candle flame 0.861±0.08candy 1 0.812±0.083candy 2 0.556±0.123coral 0.742±0.106cranberries 0.473±0.114escalator 0.74±0.098fireplace 1 0.917±0.064fish 0.63±0.111flag 0.987±0.025flag 2 0.985±0.03flames 0.843±0.085flushing water 0.541±0.114fountain 1 0.646±0.116fountain 2 0.859±0.077fur 0.535±0.105grass 1 0.761±0.099grass 2 0.7±0.107grass 3 0.887±0.074ink 0.636±0.107lava 0.441±0.118plants 0.651±0.118sea 1 0.73±0.101sea 2 0.586±0.103shiny circles 0.671±0.106shower water 1 0.809±0.082sky clouds 1 0.662±0.11sky clouds 2 0.904±0.068smoke 1 0.671±0.104smoke 2 0.6±0.119smoke 3 0.833±0.09smoke plume 1 0.767±0.097snake 1 0.797±0.089snake 2 0.738±0.107snake 3 0.77±0.096snake 4 0.724±0.101snake 5 0.885±0.071tv static 0.532±0.11underwater vegetation 1 0.594±0.116water 1 0.521±0.115water 4 0.559±0.118water 2 0.594±0.116water 3 0.685±0.107water 5 0.646±0.105waterfall 0.603±0.112waterfall 2 0.466±0.114

Long (1200-4800 ms.)0.673±0.0930.928±0.0480.723±0.090.915±0.0530.864±0.0660.716±0.0910.707±0.0980.729±0.0890.798±0.0750.712±0.0870.604±0.0930.97±0.0330.94±0.0510.688±0.0860.827±0.0730.558±0.0950.909±0.0570.971±0.0320.627±0.0940.99±0.020.971±0.0320.87±0.0630.918±0.0540.776±0.0790.947±0.0450.682±0.0970.8±0.0780.88±0.0640.941±0.0460.792±0.0790.631±0.0930.841±0.0690.917±0.0550.729±0.0940.729±0.0890.93±0.0540.68±0.0930.931±0.050.674±0.0940.637±0.0890.927±0.0490.863±0.0670.947±0.0450.896±0.0580.94±0.0470.903±0.060.95±0.0430.448±0.0990.794±0.0780.55±0.0980.806±0.0760.709±0.0880.74±0.0840.688±0.0930.767±0.0820.543±0.095

All (300-4800 ms.)0.608±0.0720.882±0.0480.702±0.0690.886±0.0470.802±0.060.665±0.0710.636±0.0720.713±0.0670.751±0.0640.69±0.0690.632±0.0720.924±0.040.876±0.050.64±0.0710.794±0.0610.522±0.0730.835±0.0550.949±0.0330.629±0.0720.989±0.0160.976±0.0230.86±0.0510.756±0.0640.727±0.0670.908±0.0430.609±0.0730.784±0.0620.806±0.0590.919±0.0410.725±0.0660.556±0.0740.771±0.0630.835±0.0560.657±0.0710.703±0.0680.869±0.050.673±0.0710.92±0.040.672±0.070.624±0.0710.892±0.0460.823±0.0570.879±0.0490.836±0.0550.868±0.050.822±0.0580.921±0.040.486±0.0740.713±0.0680.538±0.0740.708±0.0680.663±0.0710.718±0.0660.669±0.070.699±0.0680.511±0.073

Table 2: Per-texture accuracies averaged over a range of exposure times, using the concatenation layer. Each texture accuracyincludes a margin of error with a 95% statistical confidence.

Dynamic texture 300 ms.ants 0.933±0.126bamboo 1.0±0.0birds 0.895±0.138boiling water 1 0.846±0.196boiling water 2 0.808±0.151calm water 0.929±0.135calm water 2 1.0±0.0calm water 3 1.0±0.0calm water 4 0.875±0.162calm water 5 1.0±0.0calm water 6 0.913±0.115candle flame 0.944±0.106candy 1 0.765±0.202candy 2 0.864±0.143coral 0.84±0.144cranberries 0.75±0.212escalator 0.947±0.1fireplace 1 0.905±0.126fish 0.933±0.089flag 0.875±0.162flag 2 0.958±0.08flames 0.667±0.189flushing water 0.941±0.112fountain 1 0.609±0.199fountain 2 0.95±0.096fur 0.818±0.161grass 1 0.952±0.091grass 2 1.0±0.0grass 3 1.0±0.0ink 0.947±0.1lava 0.952±0.091plants 0.9±0.131sea 1 0.889±0.145sea 2 0.85±0.156shiny circles 0.808±0.151shower water 1 0.941±0.112sky clouds 1 1.0±0.0sky clouds 2 0.941±0.112smoke 1 0.867±0.172smoke 2 0.667±0.239smoke 3 1.0±0.0smoke plume 1 1.0±0.0snake 1 0.941±0.112snake 2 0.958±0.08snake 3 0.957±0.083snake 4 1.0±0.0snake 5 0.909±0.12tv static 0.684±0.209underwater vegetation 1 0.857±0.183water 1 0.929±0.135water 4 0.778±0.272water 2 0.867±0.172water 3 0.737±0.198water 5 1.0±0.0waterfall 0.947±0.1waterfall 2 0.941±0.112

400 ms.0.9±0.1860.944±0.1060.652±0.1950.895±0.1380.889±0.1190.963±0.0711.0±0.01.0±0.00.947±0.10.897±0.1111.0±0.01.0±0.00.87±0.1380.875±0.1320.957±0.0830.917±0.1111.0±0.00.765±0.2020.957±0.0831.0±0.01.0±0.00.75±0.190.88±0.1270.65±0.2091.0±0.00.95±0.0960.938±0.1190.92±0.1061.0±0.00.962±0.0741.0±0.01.0±0.01.0±0.00.857±0.1830.8±0.1750.857±0.151.0±0.01.0±0.00.773±0.1750.957±0.0831.0±0.00.958±0.081.0±0.00.917±0.1111.0±0.00.947±0.11.0±0.00.588±0.2340.958±0.080.778±0.1921.0±0.01.0±0.00.905±0.1260.944±0.1060.933±0.1260.88±0.127

600 ms.0.913±0.1151.0±0.00.933±0.1260.957±0.0830.714±0.1931.0±0.00.966±0.0660.957±0.0831.0±0.01.0±0.00.958±0.081.0±0.00.938±0.1191.0±0.01.0±0.00.926±0.0991.0±0.00.923±0.1020.944±0.1061.0±0.01.0±0.00.722±0.2070.727±0.1860.769±0.2290.947±0.11.0±0.00.917±0.1111.0±0.00.958±0.080.96±0.0770.941±0.1121.0±0.01.0±0.01.0±0.00.75±0.190.923±0.1020.947±0.10.941±0.1120.846±0.1391.0±0.01.0±0.00.964±0.0691.0±0.00.962±0.0741.0±0.01.0±0.01.0±0.00.64±0.1880.952±0.0910.952±0.0910.889±0.1190.962±0.0740.938±0.1191.0±0.00.952±0.0910.947±0.1

1200 ms.0.963±0.0711.0±0.00.947±0.10.96±0.0770.95±0.0960.962±0.0741.0±0.00.941±0.1121.0±0.00.857±0.151.0±0.01.0±0.00.905±0.1261.0±0.01.0±0.00.958±0.081.0±0.00.867±0.1720.87±0.1380.958±0.081.0±0.00.789±0.1831.0±0.00.913±0.1150.952±0.0910.955±0.0871.0±0.00.913±0.1150.923±0.1451.0±0.01.0±0.01.0±0.01.0±0.00.955±0.0870.88±0.1270.929±0.1351.0±0.01.0±0.00.889±0.1451.0±0.00.96±0.0771.0±0.01.0±0.01.0±0.01.0±0.00.957±0.0831.0±0.00.778±0.1921.0±0.00.929±0.0951.0±0.01.0±0.00.897±0.1111.0±0.00.85±0.1561.0±0.0

2400 ms.1.0±0.01.0±0.00.9±0.1310.92±0.1060.857±0.1830.952±0.0911.0±0.00.955±0.0871.0±0.01.0±0.01.0±0.01.0±0.00.846±0.1390.96±0.0770.941±0.1120.867±0.1721.0±0.00.929±0.0951.0±0.01.0±0.01.0±0.00.826±0.1550.8±0.1570.762±0.1821.0±0.01.0±0.01.0±0.00.95±0.0961.0±0.01.0±0.00.906±0.1011.0±0.01.0±0.01.0±0.00.8±0.2021.0±0.01.0±0.00.933±0.1260.944±0.1060.947±0.11.0±0.00.955±0.0871.0±0.01.0±0.01.0±0.00.95±0.0961.0±0.00.55±0.2181.0±0.00.889±0.1451.0±0.01.0±0.01.0±0.01.0±0.00.929±0.0950.773±0.175

3600 ms.1.0±0.01.0±0.00.913±0.1150.952±0.0910.889±0.1451.0±0.01.0±0.00.96±0.0771.0±0.00.944±0.1061.0±0.01.0±0.00.81±0.1680.952±0.0911.0±0.00.95±0.0961.0±0.00.947±0.11.0±0.01.0±0.01.0±0.00.917±0.1110.906±0.1010.818±0.1611.0±0.01.0±0.00.958±0.081.0±0.01.0±0.01.0±0.01.0±0.00.958±0.080.958±0.080.968±0.0620.96±0.0771.0±0.01.0±0.00.96±0.0770.929±0.0951.0±0.01.0±0.01.0±0.01.0±0.01.0±0.00.905±0.1261.0±0.01.0±0.00.76±0.1671.0±0.01.0±0.00.909±0.120.955±0.0870.875±0.1320.933±0.0890.926±0.0990.905±0.126

4800 ms.0.885±0.1231.0±0.00.966±0.0661.0±0.00.92±0.1061.0±0.00.941±0.1121.0±0.01.0±0.01.0±0.01.0±0.01.0±0.00.8±0.1570.95±0.0961.0±0.01.0±0.01.0±0.01.0±0.01.0±0.00.947±0.10.938±0.1190.842±0.1640.867±0.1720.895±0.1381.0±0.01.0±0.01.0±0.00.895±0.1381.0±0.00.813±0.1910.95±0.0960.958±0.080.889±0.1451.0±0.00.9±0.1311.0±0.01.0±0.01.0±0.00.95±0.0960.909±0.121.0±0.01.0±0.01.0±0.00.955±0.0871.0±0.01.0±0.01.0±0.00.783±0.1691.0±0.00.88±0.1271.0±0.01.0±0.00.909±0.120.962±0.0741.0±0.00.9±0.131

Table 3: Per-texture accuracies averaged over exposure times, using the flow decode layer. Each texture accuracy includes amargin of error with a 95% statistical confidence.

Dynamic texture Short (300-600 ms.)ants 0.917±0.078bamboo 0.983±0.034birds 0.807±0.102boiling water 1 0.909±0.076boiling water 2 0.811±0.089calm water 0.963±0.05calm water 2 0.986±0.027calm water 3 0.985±0.029calm water 4 0.95±0.055calm water 5 0.954±0.051calm water 6 0.956±0.049candle flame 0.986±0.028candy 1 0.857±0.092candy 2 0.912±0.067coral 0.932±0.057cranberries 0.881±0.078escalator 0.98±0.038fireplace 1 0.875±0.081fish 0.944±0.054flag 0.963±0.05flag 2 0.987±0.025flames 0.71±0.113flushing water 0.844±0.089fountain 1 0.661±0.124fountain 2 0.96±0.054fur 0.917±0.07grass 1 0.934±0.062grass 2 0.969±0.042grass 3 0.983±0.034ink 0.957±0.047lava 0.966±0.046plants 0.964±0.049sea 1 0.968±0.044sea 2 0.902±0.082shiny circles 0.788±0.099shower water 1 0.906±0.071sky clouds 1 0.985±0.029sky clouds 2 0.966±0.046smoke 1 0.825±0.094smoke 2 0.91±0.068smoke 3 1.0±0.0smoke plume 1 0.972±0.038snake 1 0.983±0.032snake 2 0.946±0.052snake 3 0.986±0.026snake 4 0.984±0.03snake 5 0.964±0.049tv static 0.639±0.121underwater vegetation 1 0.932±0.064water 1 0.887±0.085water 4 0.907±0.077water 2 0.95±0.055water 3 0.857±0.092water 5 0.981±0.037waterfall 0.945±0.06waterfall 2 0.918±0.069

Long (1200-4800 ms.)0.959±0.0391.0±0.00.934±0.0510.955±0.0440.909±0.0640.979±0.0280.988±0.0240.964±0.041.0±0.00.948±0.051.0±0.01.0±0.00.839±0.0750.963±0.0420.985±0.0290.952±0.0461.0±0.00.94±0.0510.961±0.0430.979±0.0290.985±0.0290.847±0.0770.884±0.0680.847±0.0770.989±0.0210.988±0.0230.988±0.0230.939±0.0520.989±0.0220.963±0.0410.956±0.0430.978±0.030.965±0.0390.979±0.0290.894±0.0650.988±0.0231.0±0.00.978±0.0310.929±0.0550.964±0.040.989±0.0220.986±0.0261.0±0.00.986±0.0270.973±0.0370.975±0.0341.0±0.00.721±0.0951.0±0.00.921±0.0560.978±0.030.988±0.0230.914±0.0570.969±0.0350.921±0.0560.897±0.064

All (300-4800 ms.)0.945±0.0370.993±0.0130.885±0.0510.937±0.040.861±0.0550.974±0.0260.987±0.0180.974±0.0260.979±0.0230.951±0.0360.98±0.0230.993±0.0140.846±0.0580.939±0.0390.957±0.0330.92±0.0430.993±0.0140.912±0.0460.953±0.0340.973±0.0260.986±0.0190.789±0.0660.867±0.0540.773±0.0690.979±0.0230.958±0.0330.966±0.030.952±0.0340.986±0.0190.96±0.0310.96±0.0320.972±0.0270.966±0.0290.952±0.0350.848±0.0570.952±0.0340.993±0.0140.973±0.0260.884±0.0520.94±0.0380.993±0.0130.979±0.0230.993±0.0130.966±0.030.98±0.0230.979±0.0230.986±0.0190.687±0.0750.972±0.0270.908±0.0470.952±0.0350.972±0.0270.893±0.050.973±0.0260.931±0.0420.905±0.047

Table 4: Per-texture accuracies averaged over a range of exposure times, using the flow decode layer. Each texture accuracyincludes a margin of error with a 95% statistical confidence.

Appearance group 300 ms.Regular & Near-regular 0.702±0.098Irregular 0.806±0.046Stochastic & Near-stochastic 0.616±0.03

400 ms.0.74±0.1010.853±0.0440.658±0.029

600 ms.0.838±0.0880.837±0.0430.687±0.028

1200 ms.0.84±0.0830.903±0.0360.76±0.026

2400 ms.0.954±0.0510.909±0.0370.751±0.026

3600 ms.0.878±0.0740.919±0.0310.776±0.026

4800 ms.0.827±0.0820.902±0.0350.762±0.025

Table 5: Accuracies of textures grouped by appearances, averaged over exposure times, using the concatenation layer. Eachtexture accuracy includes a margin of error with a 95% statistical confidence.

Appearance group Short (300-600 ms.)Regular & Near-regular 0.756±0.056Irregular 0.831±0.026Stochastic & Near-stochastic 0.654±0.017

Long (1200-4800 ms.)0.871±0.0380.908±0.0170.762±0.013

All (300-4800 ms.)0.821±0.0330.875±0.0150.717±0.01

Table 6: Accuracies of textures grouped by appearances, averaged over a range of exposure times, using the concatenationlayer. Each texture accuracy includes a margin of error with a 95% statistical confidence.

Appearance group 300 ms.Regular & Near-regular 0.889±0.078Irregular 0.89±0.041Stochastic & Near-stochastic 0.901±0.021

400 ms.0.933±0.0630.942±0.0310.916±0.018

600 ms.0.921±0.0670.957±0.0260.937±0.016

1200 ms.0.961±0.0430.953±0.0280.957±0.014

2400 ms.0.948±0.0570.96±0.0250.945±0.015

3600 ms.0.984±0.0310.968±0.0220.955±0.013

4800 ms.0.964±0.0490.947±0.0290.96±0.013

Table 7: Accuracies of textures grouped by appearances, averaged over exposure times, using the flow decode layer. Eachtexture accuracy includes a margin of error with a 95% statistical confidence.

Appearance group Short (300-600 ms.)Regular & Near-regular 0.914±0.04Irregular 0.93±0.019Stochastic & Near-stochastic 0.919±0.011

Long (1200-4800 ms.)0.964±0.0230.957±0.0130.954±0.007

All (300-4800 ms.)0.943±0.0220.946±0.0110.939±0.006

Table 8: Accuracies of textures grouped by appearances, averaged over a range of exposure times, using the flow decodelayer. Each texture accuracy includes a margin of error with a 95% statistical confidence.

Dynamics group 300 ms.Spatially-consistent 0.625±0.032Spatially-inconsistent 0.721±0.039

400 ms.0.664±0.0320.763±0.039

600 ms.0.698±0.030.777±0.037

1200 ms.0.741±0.0280.885±0.028

2400 ms.0.753±0.0280.854±0.032

3600 ms.0.762±0.0280.902±0.026

4800 ms.0.755±0.0280.861±0.029

Table 9: Accuracies of textures grouped by dynamics, averaged over exposure times, using the concatenation layer. Eachtexture accuracy includes a margin of error with a 95% statistical confidence.

Dynamics group Short (300-600 ms.)Spatially-consistent 0.663±0.018Spatially-inconsistent 0.753±0.022

Long (1200-4800 ms.)0.753±0.0140.876±0.015

All (300-4800 ms.)0.715±0.0110.823±0.013

Table 10: Accuracies of textures grouped by dynamics, averaged over a range of exposure times, using the concatenationlayer. Each texture accuracy includes a margin of error with a 95% statistical confidence.

Dynamics group 300 ms.Spatially-consistent 0.886±0.024Spatially-inconsistent 0.92±0.027

400 ms.0.911±0.020.942±0.023

600 ms.0.934±0.0180.949±0.021

1200 ms.0.947±0.0160.974±0.016

2400 ms.0.945±0.0160.954±0.02

3600 ms.0.955±0.0140.966±0.017

4800 ms.0.954±0.0150.964±0.018

Table 11: Accuracies of textures grouped by dynamics, averaged over exposure times, using the flow decode layer. Eachtexture accuracy includes a margin of error with a 95% statistical confidence.

Dynamics group Short (300-600 ms.)Spatially-consistent 0.911±0.012Spatially-inconsistent 0.937±0.013

Long (1200-4800 ms.)0.95±0.0080.964±0.009

All (300-4800 ms.)0.934±0.0070.953±0.008

Table 12: Accuracies of textures grouped by dynamics, averaged over a range of exposure times, using the flow decode layer.Each texture accuracy includes a margin of error with a 95% statistical confidence.

Group 300 ms.All textures 0.661±0.025

400 ms.0.699±0.025

600 ms.0.726±0.023

1200 ms.0.791±0.021

2400 ms.0.788±0.022

3600 ms.0.812±0.021

4800 ms.0.793±0.021

Table 13: Average accuracy over all textures, averaged over exposure times, using the concatenation layer. Each textureaccuracy includes a margin of error with a 95% statistical confidence.

Group Short (300-600 ms.)All textures 0.695±0.014

Long (1200-4800 ms.)0.796±0.011

All (300-4800 ms.)0.754±0.009

Table 14: Average accuracy over all textures, averaged over a range of exposure times, using the concatenation layer. Eachtexture accuracy includes a margin of error with a 95% statistical confidence.

Group 300 ms.All textures 0.898±0.018

400 ms.0.922±0.015

600 ms.0.94±0.013

1200 ms.0.956±0.012

2400 ms.0.948±0.013

3600 ms.0.959±0.011

4800 ms.0.957±0.012

Table 15: Average accuracy over all textures, averaged over exposure times, using the flow decode layer. Each textureaccuracy includes a margin of error with a 95% statistical confidence.

Group Short (300-600 ms.)All textures 0.921±0.009

Long (1200-4800 ms.)0.955±0.006

All (300-4800 ms.)0.941±0.005

Table 16: Average accuracy over all textures, averaged over a range of exposure times, using the flow decode layer. Eachtexture accuracy includes a margin of error with a 95% statistical confidence.