Learning Representations for Multimodal Data with Deep Belief Nets Nitish Srivastava [email protected]University of Toronto, Toronto, ON. M5S 3G4 Canada Ruslan Salakhutdinov [email protected]University of Toronto, Toronto, ON. M5S 3G4 Canada Abstract We propose a Deep Belief Network archi- tecture for learning a joint representation of multimodal data. The model defines a prob- ability distribution over the space of mul- timodal inputs and allows sampling from the conditional distributions over each data modality. This makes it possible for the model to create a multimodal representation even when some data modalities are missing. Our experimental results on bi-modal data consisting of images and text show that the Multimodal DBN can learn a good generative model of the joint space of image and text in- puts that is useful for filling in missing data so it can be used both for image annotation and image retrieval. We further demonstrate that using the representation discovered by the Multimodal DBN our model can signif- icantly outperform SVMs and LDA on dis- criminative tasks. 1. Introduction Information in the real world comes through multi- ple input channels. Images are associated with cap- tions and tags, videos contain visual and audio signals, sensory perception includes simultaneous inputs from visual, auditory, motor and haptic pathways. While each input modality conveys additional information, the information content of any modality is unlikely to be independent of the others. For example, images of forests and landscapes are strongly associated with tags like nature and scenery. Presented at the ICML Representation Learning Workshop, Edinburgh, Scotland, UK, 2012. Copyright 2012 by the author(s)/owner(s). Figure 1. Examples of data from the MIR Flickr Dataset, along with text generated from the Deep Belief Net by sampling from P (vtxt |vimg ,θ) The goal of this work is to learn a representation that takes this association into account. At the same time, the model must be able to handle missing data modal- ities so that the same kind of representation can be ex- tracted even when some input channels are not avail- able. One way to achieve this is by learning a joint density model over the space of multimodal inputs. Missing modalities can then be handled by sampling from the implied conditional distributions over miss- ing modalities given the observed modalities. For ex- ample, we can use a large collection of user-tagged images to learn a distribution over images and text
Transcript
Learning Representations for Multimodal Data with Deep BeliefNets
University of Toronto, Toronto, ON. M5S 3G4 Canada
Abstract
We propose a Deep Belief Network archi-tecture for learning a joint representation ofmultimodal data. The model defines a prob-ability distribution over the space of mul-timodal inputs and allows sampling fromthe conditional distributions over each datamodality. This makes it possible for themodel to create a multimodal representationeven when some data modalities are missing.Our experimental results on bi-modal dataconsisting of images and text show that theMultimodal DBN can learn a good generativemodel of the joint space of image and text in-puts that is useful for filling in missing dataso it can be used both for image annotationand image retrieval. We further demonstratethat using the representation discovered bythe Multimodal DBN our model can signif-icantly outperform SVMs and LDA on dis-criminative tasks.
1. Introduction
Information in the real world comes through multi-ple input channels. Images are associated with cap-tions and tags, videos contain visual and audio signals,sensory perception includes simultaneous inputs fromvisual, auditory, motor and haptic pathways. Whileeach input modality conveys additional information,the information content of any modality is unlikely tobe independent of the others. For example, imagesof forests and landscapes are strongly associated withtags like nature and scenery.
Presented at the ICML Representation Learning Workshop,Edinburgh, Scotland, UK, 2012. Copyright 2012 by theauthor(s)/owner(s).
Figure 1. Examples of data from the MIR Flickr Dataset,along with text generated from the Deep Belief Net bysampling from P (vtxt|vimg, θ)
The goal of this work is to learn a representation thattakes this association into account. At the same time,the model must be able to handle missing data modal-ities so that the same kind of representation can be ex-tracted even when some input channels are not avail-able. One way to achieve this is by learning a jointdensity model over the space of multimodal inputs.Missing modalities can then be handled by samplingfrom the implied conditional distributions over miss-ing modalities given the observed modalities. For ex-ample, we can use a large collection of user-taggedimages to learn a distribution over images and text
Multimodal Learning with Deep Belief Nets
P (vimg, vtxt|θ) such that it is easy to sample fromP (vtxt|vimg, θ) and from P (vimg|vtxt, θ) so that we cando image annotation (Figure 1) and image retrieval(Figure 2). In addition, it is also desirable that therepresentation be useful for discriminative tasks, suchas object recognition.
Before we describe our model in detail, it is useful tonote why such a model is required. In many appli-cations, observations come from different input chan-nels each of which has a different representation andcorrelational structure. For example, text is usuallyrepresented as sparse word count vectors whereas animage is represented using pixel intensities or outputsof feature extractors which are real-valued and dense.This makes it much harder to discover relationshipsacross modalities than relationships among features ofthe same modality. There is a lot of structure in theinput but it is difficult to discover the highly non-linearrelationships that exist between features across differ-ent modalities. Moreover, these observations are noisyand may have missing values. Using our probabilisticmodel, it will be possible to discover joint latent rep-resentations that capture relationships across variousmodalities. Different modalities typically carry differ-ent kinds of information. For example, people oftencaption an image to say things that may not be ob-vious from the image itself, such as the name of theperson or place in the picture. It would not be possibleto discover a lot of useful information about the worldunless we do multimodal learning.
In this paper, we propose a model based on Deep BeliefNets (Hinton & Salakhutdinov, 2006). The key idea isto first use separate modality-friendly latent variablemodels to learn low-level representations of each datamodality independently. For doing this we can lever-age a large supply of unlabeled data to separately learngood generative models for each modality. Indeed, formany domains, including text retrieval, speech per-ception, and machine vision, unlabeled data is readilyavailable. While the inputs to each of these separatemodels will typically belong to different modalities, ourmodel will learn latent representations that are similarin form and correlational structure. The latent repre-sentations for different modalities can then be concate-nated to form a multimodal input. Higher-order latentvariables can then be used to model the distributionover this input. The posteriors over the higher-ordervariables can then be used to represent the multimodalinput.
There have been several approaches to learning frommultimodal data. In particular, Huiskes et al. (2010)showed that using captions, or tags, in addition to
Figure 2. Examples of images retrieved using featuresgenerated from a Deep Belief Net by sampling fromP (vimg|vtxt, θ)
standard low-level image features significantly im-proves classification accuracy of SVM and LDA (Lin-ear Discriminant Analysis) models. A similar ap-proach of Guillaumin et al. (2010), based on multi-ple kernel learning framework, further demonstratedthat an additional text modality can improve the ac-curacy of SVMs on various object recognition tasks.However, all of these approaches are discriminative bynature and cannot make use of large amounts of unla-beled data or deal easily with noisy or missing inputmodalities.
On the generative side, Xing et al. (2005) used dual-wing harmoniums to build a joint model of images andtext, which can be viewed as a linear RBM modelwith Gaussian hidden units together with Gaussianand Poisson visible units. Most similar to our work isthe recent approach of Ngiam et al. (2011) that used adeep autoencoder for speech and vision fusion. Thereare, however, several crucial differences. First, in thiswork we focus on integrating together very differentdata modalities: sparse word count vectors, and real-
Multimodal Learning with Deep Belief Nets
valued dense image features. Second, we develop aDeep Belief Network as a generative model as opposedto unrolling the network and finetuning it as an au-toencoder. While both approaches have lead to inter-esting results in several domains, using a generativemodel is important here as it allows our model to eas-ily handle missing data modalities.
2. Background: RBMs and TheirGeneralizations
2.1. Restricted Boltzmann Machines
A Restricted Boltzmann Machine is an undirectedgraphical model with visible units v ∈ {0, 1}D andhidden units h ∈ {0, 1}F with each visible unit con-nected to each hidden unit. The model defines an en-ergy function E : {0, 1}D+F → R
E(v, h; θ) = −a>v − b>h− v>Wh,
where θ = {a,b,W} are the model parameters. Thejoint distribution over the visible and hidden units isdefined by:
P (v,h; θ) =1
Z(θ)exp (−E(v,h; θ)), (1)
where Z(θ) is the normalizing constant.
2.2. Gaussian RBM
Consider modelling visible real-valued units v ∈ RD
and let h ∈ {0, 1}F be binary stochastic hidden units.The energy of the state {v,h} of the Gaussian RBMis defined as follows:
E(v,h; θ) =
D∑i=1
(vi − bi)2
2σ2i
−D∑i=1
F∑j=1
viσiWijhj−
F∑j=1
ajhj ,
where θ = {a,b,W, σ} are the model parameters.This leads to the following conditional distribution:
P (vi|h; θ) = N
bi + σi
F∑j=1
Wijhj , σ2i
(2)
2.3. Replicated Softmax Model
The Replicated Softmax Model Salakhutdinov & Hin-ton (2009) is useful for modelling sparse count data,such as word count vectors in a document. Let v ∈ NK
be a vector of visible units where vk counts the num-ber of times word k occurs in the document with thevocabulary of size K. Let h ∈ {0, 1}J be binarystochastic hidden topic features. The energy of thestate {v,h} is defined as follows
E(v, h; θ) = −K∑
k=1
J∑j=1
Wkjhjvk−K∑
k=1
vkbk−MJ∑
j=1
hjaj
where θ = {a,b,W} are the model parameters andM =
∑k vk is the total number of words in a docu-
ment. The leads to the following conditional distribu-tion:
P (vk = 1|h; θ) =exp(−bk +
∑Jj=1Wkjhj)∑K
k′=1 exp(−bk′ +∑J
j=1Wk′jhj)(3)
For all of the above models, exact maximum likelihoodlearning is intractable. In practice, efficient learning isperformed by following an approximation to the gradi-ent of the Contrastive Divergence (CD) objective (Hin-ton, 2002).
3. Multimodal Deep Belief Network
We illustrate the construction of a multimodal DBNusing an image-text bi-modal DBN as our running ex-ample. Let vm ∈ RD denote an image and vt ∈ NK
denote a text input. Consider modelling each datamodality using a separate two-layer DBN (see Fig. 3).The probability that each DBN model assigns to a vis-ible vector is:
P (vm) =∑
h(1),h(2)
P (h(2),h(1))P (vm|h(1)) (4)
P (vt) =∑
h(1),h(2)
P (h(2),h(1))P (vt|h(1)) (5)
The image-specific DBN uses Gaussian RBM tomodel the distribution over real-valued image features,whereas text-specific DBN uses Replicated Softmaxesto model the distribution over word count vectors. Theconditional probabilities of the visibles given hiddensused in Eqs 4, 5 are as shown in Eqs 2, 3 respectively.
To form a multimodal DBN, we combine the two mod-els by learning a joint RBM on top of them. The re-sulting graphical model is shown in Fig. 3, right panel.The joint distribution can be written as:
P (vm,vt) =∑
h(2)m ,h
(2)t ,h(3)
P (h(2)m ,h
(2)t ,h(3))×
∑h
(1)m
P (vm|h(1)m )P (h(1)
m |h(2)m )×
∑h
(1)t
P (vt|h(1)t )P (h
(1)t |h
(2)t ). (6)
The parameters of this mulitmodal DBN can belearned approximately by greedy layer-wise trainingusing CD.
Note that the Multimodal DBN can be described as acomposition of unimodal pathways. Each pathway islearned separately in a completely unsupervised fash-ion, which allows us to leverage a large supply of un-labeled data. Any number of pathways each with any
Multimodal Learning with Deep Belief Nets
Image-specificDBN
Text-specificDBN
Multimodal DBN
Figure 3. Left: Image-specific two-layer DBN that uses a Gaussian model to model the distribution over real-valuedimage features. Middle: Text-specific two-layer DBN that uses a Replicated Softmax model to model its distributionover the word count vectors. Right: A Multimodal DBN that models the joint distribution over image and text inputs.
number of layers could potentially be used. The typeof lower RBMs in each layer could be different, ac-counting for different kinds of input distributions, aslong as the final hidden representations at the end ofeach pathway are of the same type.
The intuition behind our model is as follows. Eachdata modality may have very different statistical prop-erties which make it difficult for a shallow model to di-rectly find correlations across modalities. The purposeof the independent modality-friendly models (Eq 4, 5)is to learn higher-level representations that removesuch modality-specific correlations so that the top levelRBM is presented with features that are relatively“modality-free”, i.e., they are more alike in termsof their statistical properties than the original inputswere. In other words, given the original inputs, it iseasy to say which represents images and which repre-sents text using their sparsity and correlational struc-ture. But, looking at the higher-level hidden featuresin the DBNs, it is more difficult to make such a dis-tinction. Hence, the top-level joint RBM can pick upcross-modal relationships easily.
3.1. Generative Tasks
As argued in the introduction, many real-world appli-cations will often have one or more of its modalitiesmissing. We can infer missing values by drawing sam-ples from the conditional model, which would allow usto properly use all input channels.
As an example, consider generating text conditionedon a given image1 vm. We first infer the values of the
hidden variables h(2)m in the image pathway by forward
propagating vm through to the last hidden layer. Con-
ditioned on h(2)m at the top level RBM, we can perform
alternating Gibbs sampling using the following condi-
1Generating image features conditioned on text can bedone in a similar way.
tional distributions:
P (h(3)|h(2)m ,h
(2)t ) = σ
(W (3)
m h(2)m +W
(3)t h
(2)t + b
),(7)
P (h(2)t |h(3)) = σ
(W
(3)>t h(3) + at
), (8)
where σ(x) = 1/(1 + e−x). The sample h(2)t can then
be propagated back through the text pathway to gen-erate a distribution over the softmax vocabulary. Thisdistribution can then be used to sample words.
3.2. Discriminative Tasks
The model can also be used for classification tasks byadding a discriminative layer of weights on top of theMultimodal DBN and finetuning the network to opti-mize a cross-entropy objective. In our experiments weuse a simple logistic classifier to do 1-vs-all classifica-tion and finetune the model with stochastic gradientdescent.
4. Experiments
4.1. Dataset and Feature Extraction
The MIR Flickr Data set (Huiskes & Lew, 2008) wasused in our experiments. The data set consists of1 million images retrieved from the social photogra-phy website Flickr along with their user assigned tags.The collection includes images released under the Cre-ative Commons License. Among the 1 million images,25,000 have been annotated for 24 concepts includingobject categories such as bird, tree, people and scenecategories like indoor, sky and night. For 14 of them,a stricter labelling was done in which an image wasassigned an annotation only if the corresponding cate-gory was salient in the image. This leads to a total of38 classes. Each image may belong to several classes.The unlabeled 975,000 images were used only for pre-training the DBN. We use 15,000 images for trainingand 10,000 for testing, following (Huiskes et al., 2010).Mean Average Precision (MAP) is used as the perfor-
Multimodal Learning with Deep Belief Nets
mance metric. Results are averaged over 10 randomsplits of training and test sets.
There are more than 800,000 distinct tags in thedataset. In order to keep the text representation man-ageable, each text input was represented using a vo-cabulary of the 2000 most frequent tags. After re-stricting to this vocabulary, the average number oftags associated with an image is 5.15 with a stan-dard deviation of 5.13. There are 128,501 imageswhich do not have any tags out of which 4,551 arein the labelled set. Hence about 18% of the labelleddata does not have any tags. Word counts w were re-placed with dlog(1 + w)e. We concatenated PyramidHistogram of Words (PHOW) features (Bosch et al.,2007), Gist (Oliva & Torralba, 2001) and MPEG-7 de-scriptors (Manjunath et al., 2001) (EHD, HTD, CSD,CLD, SCD) to get a 3857 dimensional representa-tion of images. Each dimension was mean-centered.PHOW features are bags of image words obtained byextracting dense SIFT features over multiple scalesand clustering them.
4.2. Model Architecture and Learning
The image pathway consists of a Gaussian RBM with3857 visible and 1000 hidden units, followed by anotherlayer of 1000 hidden units. The text pathway consistsof a Replicated Softmax Model with 2000 visible and1000 hidden units followed by another layer of 1000hidden units. The joint layer also contains 1000 hiddenunits. The model was not found to be very sensitiveto the choice of these hyperparameters.
We pretrained each pathway with greedy layer-wiseCD1. The variance of each Gaussian unit was fixedto be its empirical variance in the training set. Fordiscriminative tasks, we perform 1-vs-all classificationusing logistic regression on the last layer of hiddenunits in the joint model. The entire network was fine-tuned with stochastic gradient descent for each of the38 classes separately since the class labels overlap. Wesplit the 15K training set into 10,000 for training and5,000 for validation.
4.3. Discriminative Aspect
In our first set of experiments, we evaluate the multi-modal DBN as a discriminative model. Table 1 showsthe results of our comparison with Linear Discrimi-nant Analysis (LDA) and Support Vector Machines(SVMs) (Huiskes et al., 2010). The LDA and SVMmodels were trained using the labelled data on con-catenated image and text features. Moreover, SIFT-based features were not used. Hence, to make a faircomparison, we first trained our model without us-
ing unlabeled data and using a similar set of features(i.e., excluding our SIFT-based features). We call thismodel DBN-Lab. Table 1 shows that the DBN-Labmodel already outperforms its competitor SVM andLDA models across many classes. DBN-Lab achievesa MAP (mean Average Precision over 38 classes) of0.503. This is compared to 0.475 and 0.492 achievedby SVM and LDA models.
To quantify the effect of using unlabeled data, wenext trained a Multimodal DBN that used all of975,000 unlabeled examples. We call this modelDBN-Unlab. The only difference between the DBN-Unlab and DBN-Lab models is that DBN-Unlab usedunlabeled data during its pretraining stage. The in-put representation for both models remained the same.Not surprisingly, the DBN-Unlab model significantlyimproved upon DBN-Lab almost across all classes,achieving a MAP of 0.532. Next, we trained a thirdmodel, called DBN, that used SIFT-based featuresalong with unlabeled data. Table 1 shows that usingSIFT features provided additional gains in model per-formance, achieving a MAP of 0.563.
We also compare to an autoencoder that was initial-ized with the DBN weights and finetuned as proposedin Ngiam et al. (2011) AUTOENCODER. It per-forms much better than SVM and LDA getting a MAPof 0.547. It does better than the DBN model on somecategories, however, on average it does not do as well.Notice that the autoencoder model does quite well onobject-level categories such as bird, car and food.
There are several scenarios in which one may wantto use the multimodal DBN for classification. Thesimplest is the case where images and associated tagsare available for both training and testing. However,it is often the case that some training and test casesmay not have tags at all. For example, in our setting,18% of the labelled data has no text input. One wayto deal with this problem is to simply use a text inputof 0 in cases where there are no tags. All the modelsdiscussed till now correspond to this scenario. i.e., thetraining and test sets are used as given, (with a zerotext input when no tags are present).There is an alternative way of dealing with missingtext. The generative model defined by the DBN canbe used to infer a text input conditioned on the imageinput. This reconstructed text can then be used tofill in the missing text. To see whether this methodof completing missing data is useful for classification,we train discriminative models using the training setas given but at test time, missing text data is filled inusing the method described in section 3.1. We call thismodel DBN-Recon. Mean-field inference was used in
Multimodal Learning with Deep Belief Nets
Table 1. Comparison of AP scores of various Mutlimodal DBNs with SVM and LDA models on the MIR Flickr Dataset.
Figure 4. Visual comparison of LDA, SVM, Autoencoder,DBN and DBN-Recon models from Table 1
place of Gibbs Sampling to reduce noise. Table 1 showsthat on average, the DBN-Recon model slightly out-performs the DBN model, achieving an average MAPof 0.566 compared to DBN’s 0.563. Our best modelsgive significant improvements over SVMs and LDA foralmost all classes. For some classes they outperformthem by a very large margin e.g., class sea* goes from0.201 (SVM) to 0.419 (DBN-Recon), tree* from 0.321to 0.546 and clouds* from 0.434 to 0.739). Figure 4shows the difference in AP scores of all the models inTable 1 with respect to the SVM model. The DBNand DBN-Recon curves outperform other models over
majority of classes.
4.4. Multimodal Aspect
While the above experiments showed that DBNs out-perform other multimodal methods, it is not obvi-ous that learning multimodal features helps over us-ing only one input modality. In this set of experi-ments, we focus on evaluating the ability of our modelto learn multimodal features that are better for dis-criminative tasks than unimodal features. In Table 2we compare our model with an SVM over Image fea-tures alone (Image-SVM) (Huiskes et al., 2010), aDBN over image features alone (Image-DBN) and aDBN over text features alone (Text-DBN). The uni-modal DBNs were constructed by adding one extralayer to the unimodal pathways used for the multi-modal DBNs, so that the number of hidden layers inall the DBNs is the same. The best multimodal DBN(DBN-Recon) clearly achieves far better overall per-formance. However, one may not find this to be veryimpressive given that the multimodal model had moredata available to it at test time than any of the othermodels which used either image or text features only.
Therefore, to make a fair comparison, we conductedthe following experiment. We take a multimodal DBNmodel that was pretrained and finetuned with bothimage and text features. However, at test time onlyimage features are provided as input and the text inputis replaced by zeros. This model is shown as DBN-
Multimodal Learning with Deep Belief Nets
Table 2. Evaluation of the multimodal aspect of the model. Multimodal DBNs outperform unimodal models even whenonly one modality is given at test time.
NoText in Table 2. Observe that the DBN-NoTextmodel performs significantly better than both SVMand DBN image only models. This result suggests thatlearning multimodal features helps even when somemodalities are absent at test time. Having multiplemodalities regularizes the model and makes it learnmuch better features. Moreover, this means that wedo not need to learn separate models to handle eachpossible combination of missing data modalities. Onejoint model can be deployed at test time and used forany situation that may arise.
We can further improve performance if missing textinput is inferred using the generative model and pro-vided as input to the discriminative model at test time.This model is shown as DBN-NoText-Recon. Fig-ure 5 shows the difference in AP scores of all themodels in Table 2 with respect to an Image-SVM.The DBN-Recon curve outperforms other models overall classes. The DBNs that use only unimodal in-puts (DBN-NoText and DBN-NoText-Recon) do bet-ter than other unimodal models.
4.5. Generative Aspect
To evaluate the generative aspect of our model qual-itatively, we look at samples of text generated fromthe multimodal DBN by conditioning on images takenfrom the test set. The images were chosen so as tocover a large number of the 38 categories. They areshown along with generated text in Figure 6. Themodel is extremely good at inferring text for imagesbelonging to scene level categories such as clouds,night*, sea*, and water. Looking at the AP scores
Figure 5. Visual comparison of models in Table 2
in Table 1 and comparing DBN-Recon with DBN, wesee that for these classes significant gains in AP scoreswere made, e.g., sea* goes from 0.359 to 0.419 (a rel-ative improvement of 16%). For finer categories likefood and transport it does not help improve classifica-tion accuracy.
We also look at images that were retrieved based onfeatures generated from the model conditioned on text.Figure 2 shows some results where we retrieve imagesfrom a subset of the test set consisting of 4000 ran-domly chosen images. We start with a manually cho-sen piece of text and infer image features conditionedon it. Then we find the nearest neighbors to these fea-tures and retrieve the corresponding images. We usedthe L2 distance between the feature vectors to findnearest neighbors where all features were normalizedto have zero mean and unit variance.
Multimodal Learning with Deep Belief Nets
Figure 6. Examples of text generated by the DBN conditioned on images
5. Conclusions and Future WorkWe proposed a Deep Belief Network architecture forlearning multimodal data representations. The modelfuses multiple data modalities into a joint hidden rep-resentation. The model defines a joint density modelover multimodal input space that can be used for fill-ing in missing inputs. It also performs well on dis-criminative tasks. When only one data modality ispresent at test time, it fills in the missing data and per-forms better than unimodal models which were trainedon one modality alone. Qualitative evaluation of themodel for image annotation and retrieval suggests thatit learns meaningful conditional distributions. Largeamounts of unlabeled data can be effectively utilizedby the model. Pathways for each modality can betrained independently and “plugged in” together whenlearning multimodal features.
Our method benefits from the fact that the statisticalproperties of the final hidden representations across allpathways are similar. However, we did not explicitlyimpose any explicit objective to achieve this. It wouldbe interesting to explore how this method can be im-proved by having an explicit penalty or constraint oncertain properties of the hidden representations suchas sparsity and entropy.
References
Bosch, A, Zisserman, Andrew, and Munoz, X. Image clas-sification using random forests and ferns. IEEE 11thInternational Conference on Computer Vision (2007),23:1–8, 2007.
Guillaumin, M., Verbeek, J., and Schmid, C. Multi-modal semi-supervised learning for image classification.
In Computer Vision and Pattern Recognition (CVPR),2010 IEEE Conference on, pp. 902 –909, june 2010.
Hinton, Geoffrey and Salakhutdinov, Ruslan. Reducing thedimensionality of data with neural networks. Science,313(5786):504 – 507, 2006.
Hinton, Geoffrey E. Training products of experts by min-imizing contrastive divergence. Neural Computation, 14(8):1711–1800, 2002.
Huiskes, Mark J. and Lew, Michael S. The MIR Flickr re-trieval evaluation. In MIR ’08: Proceedings of the 2008ACM International Conference on Multimedia Informa-tion Retrieval, New York, NY, USA, 2008. ACM.
Huiskes, Mark J., Thomee, Bart, and Lew, Michael S. Newtrends and ideas in visual concept detection: the MIRflickr retrieval evaluation initiative. In Multimedia In-formation Retrieval, pp. 527–536, 2010.
Manjunath, B.S., Ohm, J.-R., Vasudevan, V.V., and Ya-mada, A. Color and texture descriptors. Circuits andSystems for Video Technology, IEEE Transactions on,11(6):703 –715, 2001.
Ngiam, Jiquan, Khosla, Aditya, Kim, Mingyu, Nam,Juhan, Lee, Honglak, and Ng, Andrew Y. Multimodaldeep learning. In International Conference on MachineLearning (ICML), Bellevue, USA, June 2011.
Oliva, Aude and Torralba, Antonio. Modeling the shapeof the scene: A holistic representation of the spatial en-velope. International Journal of Computer Vision, 42:145–175, 2001.
Salakhutdinov, Ruslan and Hinton, Geoffrey E. Replicatedsoftmax: an undirected topic model. In NIPS, pp. 1607–1614. Curran Associates, Inc., 2009.
Xing, Eric P., Yan, Rong, and Hauptmann, Alexander G.Mining associated text and images with dual-wing har-moniums. In UAI, pp. 633–641. AUAI Press, 2005.
