IEEE TRANSACTIONS ON MULTIMEDIA 1 Refining Image Categorization by Exploiting Web Images and General Corpus Yazhou Yao, Student Member, IEEE, Jian Zhang, Senior Member, IEEE, Fumin Shen, Member, IEEE, Xiansheng Hua, Fellow, IEEE, Wankou Yang, Member, IEEE, and Zhenmin Tang Abstract—Studies show that refining real-world categories into semantic subcategories contributes to better image modeling and classification. Previous image sub-categorization work relying on labeled images and WordNet’s hierarchy is not only labor- intensive, but also restricted to classify images into NOUN subcategories. To tackle these problems, in this work, we exploit general corpus information to automatically select and subse- quently classify web images into semantic rich (sub-)categories. The following two major challenges are well studied: 1) noise in the labels of subcategories derived from the general corpus; 2) noise in the labels of images retrieved from the web. Specifically, we first obtain the semantic refinement subcategories from the text perspective and remove the noise by the relevance-based approach. To suppress the search error induced noisy images, we then formulate image selection and classifier learning as a multi-class multi-instance learning problem and propose to solve the employed problem by the cutting-plane algorithm. The experiments show significant performance gains by using the generated data of our way on both image categorization and sub-categorization tasks. The proposed approach also consistently outperforms existing weakly supervised and web-supervised ap- proaches. Index Terms—General corpus information, image categoriza- tion, sub-categorization, web-supervised I. I NTRODUCTION I Mage categorization has achieved a great progress in the past few years, but it still needs a massive amount of manually labeled data [2], [3], [4], [5], [6]. Meanwhile, image sub-categorization has been used to improve performance in a wide variety of vision tasks. For example, object detection [12], animal behaviour analysis [18] and image classification [1]. Subdividing categories into subcategories multiples the number of labels, aggravating the annotation problem. With the development of Internet, the number of digital images is growing extremely rapidly. How to effectively categorize these images has become an increasingly serious problem. Further, previously defined NOUN categories usually unable to have a good description for these emerging images, which have a variable appearance, positions, and poses [31], [45]. The categories need to be divided into more semantic rich subcategories to cover the finer semantic differences. ImageNet [19] is an image dataset organized according to the Y. Yao and J. Zhang are with the Global Big Data Technologies Center, University of Technology Sydney, Australia. F. Shen is with the School of Computer Science and Engineering, University of Electronic Science and Technology of China. X. Hua is a researcher/senior director in Alibaba Group, China. W. Yang is with the School of Automation, SouthEast University, China Z. Tang is with the School of Computer Science and Engineering, Nanjing University of Science and Technology, China. helicopters airplane pilotless airplane corsair airplane F16 airplane gulfstream airplane danish dog golden dog akita dog charles dog airedale dog Sub-categories rearing horse riding horse running horse grazing horse jumping horse Categories peacock bird goose bird parrot bird kingfisher Bird buzzard bird Fig. 1: Image categorization and sub-categorization with the vertical axis representing different categories and the horizon- tal axis representing the different subcategories of the same category. WordNet [44] hierarchy. It provides the research community not only with thousands of categories and millions of images, but also with refinement labels in a hierarchy. However, the process of constructing ImageNet is both time-consuming and labor-intensive. For example, it has taken several years to construct the ImageNet. In addition, the ImageNet requires the pre-existing expert knowledge WordNet and it only contains the NOUN subcategories (e.g., category “airplane”, “dog” and “bird” in Fig. 1), it does not contain the VERB subcategories (e.g., category “horse” in Fig. 1). To reduce the cost of manual annotation, automatic methods by exploiting web images for image categorization [9], [21], [52] have attracted more and more people’s attention. Fergus et al. [33] took the probabilistic latent semantic analysis (pLSA) technique to automatically select and learn object categories from web images. Hua et al. [21] proposed the use of a clustering based method to filter “group” noisy images and a propagation-based method to filter individual noisy images. Niu et al. [52] proposed to set each latent domain as a “bag” and the images therein as “instances”, then image selection and classifier learning are formulated as a multi-instance learning problem. The advantage of these methods is that the need for manual intervention is eliminated. Unlike these studies, our proposed work simultaneously addresses the issues of image categorization and sub-categorization by levering the general corpus and web images. arXiv:1703.05451v1 [cs.MM] 16 Mar 2017
Transcript
IEEE TRANSACTIONS ON MULTIMEDIA 1
Refining Image Categorization by Exploiting WebImages and General Corpus
Abstract—Studies show that refining real-world categories intosemantic subcategories contributes to better image modeling andclassification. Previous image sub-categorization work relyingon labeled images and WordNet’s hierarchy is not only labor-intensive, but also restricted to classify images into NOUNsubcategories. To tackle these problems, in this work, we exploitgeneral corpus information to automatically select and subse-quently classify web images into semantic rich (sub-)categories.The following two major challenges are well studied: 1) noise inthe labels of subcategories derived from the general corpus; 2)noise in the labels of images retrieved from the web. Specifically,we first obtain the semantic refinement subcategories from thetext perspective and remove the noise by the relevance-basedapproach. To suppress the search error induced noisy images,we then formulate image selection and classifier learning asa multi-class multi-instance learning problem and propose tosolve the employed problem by the cutting-plane algorithm. Theexperiments show significant performance gains by using thegenerated data of our way on both image categorization andsub-categorization tasks. The proposed approach also consistentlyoutperforms existing weakly supervised and web-supervised ap-proaches.
Index Terms—General corpus information, image categoriza-tion, sub-categorization, web-supervised
I. INTRODUCTION
IMage categorization has achieved a great progress in thepast few years, but it still needs a massive amount of
manually labeled data [2], [3], [4], [5], [6]. Meanwhile, imagesub-categorization has been used to improve performance ina wide variety of vision tasks. For example, object detection[12], animal behaviour analysis [18] and image classification[1]. Subdividing categories into subcategories multiples thenumber of labels, aggravating the annotation problem.
With the development of Internet, the number of digitalimages is growing extremely rapidly. How to effectivelycategorize these images has become an increasingly seriousproblem. Further, previously defined NOUN categories usuallyunable to have a good description for these emerging images,which have a variable appearance, positions, and poses [31],[45]. The categories need to be divided into more semanticrich subcategories to cover the finer semantic differences.ImageNet [19] is an image dataset organized according to the
Y. Yao and J. Zhang are with the Global Big Data Technologies Center,University of Technology Sydney, Australia.
F. Shen is with the School of Computer Science and Engineering, Universityof Electronic Science and Technology of China.
X. Hua is a researcher/senior director in Alibaba Group, China.W. Yang is with the School of Automation, SouthEast University, ChinaZ. Tang is with the School of Computer Science and Engineering, Nanjing
Fig. 1: Image categorization and sub-categorization with thevertical axis representing different categories and the horizon-tal axis representing the different subcategories of the samecategory.
WordNet [44] hierarchy. It provides the research communitynot only with thousands of categories and millions of images,but also with refinement labels in a hierarchy. However, theprocess of constructing ImageNet is both time-consuming andlabor-intensive. For example, it has taken several years toconstruct the ImageNet. In addition, the ImageNet requires thepre-existing expert knowledge WordNet and it only containsthe NOUN subcategories (e.g., category “airplane”, “dog” and“bird” in Fig. 1), it does not contain the VERB subcategories(e.g., category “horse” in Fig. 1).
To reduce the cost of manual annotation, automatic methodsby exploiting web images for image categorization [9], [21],[52] have attracted more and more people’s attention. Fergus etal. [33] took the probabilistic latent semantic analysis (pLSA)technique to automatically select and learn object categoriesfrom web images. Hua et al. [21] proposed the use of aclustering based method to filter “group” noisy images anda propagation-based method to filter individual noisy images.Niu et al. [52] proposed to set each latent domain as a “bag”and the images therein as “instances”, then image selection andclassifier learning are formulated as a multi-instance learningproblem. The advantage of these methods is that the need formanual intervention is eliminated. Unlike these studies, ourproposed work simultaneously addresses the issues of imagecategorization and sub-categorization by levering the generalcorpus and web images.
arX
iv:1
703.
0545
1v1
[cs
.MM
] 1
6 M
ar 2
017
IEEE TRANSACTIONS ON MULTIMEDIA 2
There are also few previous works [1], [31], [45], [48]dealing with the image sub-categorization problem. Mansur etal. [1] proposed using probabilistic Latent Semantic Analysis(pLSA) to find subcategories and these subcategories arebased on the similarity of images. In [31], both of positiveand negative images are used to learn subcategories. Par-ticularly, a new model by joint clustering and classificationwas proposed for discriminative sub-categorization. Wanget al. [48] designed a formulation for dictionary learning(subcategories) by maximizing classification margins MIL.However, as some of the previous work [8], methods [1],[31], [48] still cannot assign semantic refinement labels forthe newly discovered subcategories. Ristin et al. [45] adoptthe framework of Random Forests and proposed a regularizedobjective function that takes into account relations betweencategories and subcategories to improve the classification ofsubcategories. Unlike previous works, method [45] can clas-sify images into subcategories, but only NOUN subcategories.It is not comprehensive enough for describing the refinementimages (e.g., category “horse” in Fig. 1). In our work, weexploit general corpus information and web images for imagecategorization and sub-categorization. Our proposed approachcan not only classify images into NOUN subcategories, butalso into VERB, ADJECTIVE and ADVERB subcategories.
Motivated by the situation described above, we proposea novel automatically web-supervised image categorizationand sub-categorization framework. In our work, we mainlyconsider the following two important issues: 1) the labels ofsubcategories derived from the general corpus usually havenoise, how can we select useful labels of subcategories fromthese noisy labels; 2) the retrieved web images are oftenassociated with inaccurate labels, so the learnt classifiersmay be less robust, and the classification performance maybe significantly degraded as well, how can we select usefulimages and learn domain robust classifiers from these noisyweb training images.
To find the labels of semantic refinement subcategories, wesearch the given categories in Google Books Ngram Corpus(GBNC) [16] with Parts-Of-Speech (POS), specifically withNOUN, VERB, ADJECTIVE and ADVERB. Further, as thelabels of subcategories derived from the general corpus tend tohave noise, we apply a relevance-based approach for removingnoise and selecting the useful labels of subcategories. Finally,to cope with label noise of web training images, we treat eachselected subcategory as a “bag” and the images therein as“instances”. In specific, we propose a new multi-class MILformulation to select the images from each bag and learn theclassifiers for the categories and subcategories. Our aim isto select a subset of images from each bag to represent thisbag, such that the training bags from all the categories canbe well separated. To verify the superiority of our proposedapproach, we conducted experiments on both image catego-rization and sub-categorization tasks. The experimental resultsdemonstrated the superiority of our proposed approach.
The main contributions of this work are summarized asfollows:
1) Compared to existing methods, our proposed frameworkcan not only classify images into NOUN subcategories,
but also into VERB, ADVERB and ADJECTIVE subcat-egories. Our proposed framework has a better semanticrefinement descriptions for the categories.
2) To suppress the search error and noisy subcategories(which are not filtered out) induced noisy images, weformulate image selection and classifier learning as amulti-class multi-instance learning problem and proposeto solve the employed problem by the cutting-planealgorithm.
3) We propose a new unified objective function to jointlylearn the classifiers for categories and subcategories. Ourproposed formulation can not only consider the relation-ship between category and its subcategories, but alsoconsider the relationship between different categories.Thus, the classifiers in our work have a better domainadaptation ability.
The rest of the paper is organized as follows. In SectionII, a brief discussion of related works is given. SectionIII elaborates the proposed framework with the optimizationalgorithm. The experimental evaluations and discussions arepresented in Section IV. Lastly, the conclusion and futureresearch directions are offered in Section V.
II. RELATED WORK
Due to the emergence of ImageNet, deep convolutionalneural networks (CNN) have achieved a great success inimage categorization. However, deep CNNs are computation-ally intensive and require a large number of labeled data.Simpler classifiers like support vector machine (SVM) [50]and nearest class mean classifiers (NCM) [46] provide uswith another alternative which have much shorter running timeand acceptable classification accuracy. To reduce the cost ofmanual annotation, some of the previous works also concen-trated on the task of “cleaning up” web images for trainingdata collection. For example, Fergus et al. [28] proposed theuse of visual classifiers learned from Google image searchengine to re-rank the images based on the visual consistency.Subsequent methods [27], [25], [20] have employed similarremoving mechanisms to automatically construct clean imagedatasets for training classifiers. In our work, we focus onanother fundamental, yet often ignored, aspect of the problem:we argue that the current poor performance of classificationmodels learned from the web is due to the selected imageswhich may have different distributions with the test images.
Our work is also related to the recent works for latentdomains discovering methods. In [32], Hoffman et al. proposedusing a hierarchical clustering technique to find the feasiblelatent domains while [30] adopt maximum distinctiveness andmaximum learnability for different latent domains separation.Xiong et al. [22] proposed a squared-loss mutual informationbased clustering model with category distribution priority ineach domain to infer the domain assignment for images. Thedifference between our work and [32], [30], [22] is methods[32], [30], [22] cannot assign the semantic refinement labelsto newly discovered latent domains. All the discovered latentdomains still only have the label of the coarse category. Incontrast, our work aims to classify images into categories and
IEEE TRANSACTIONS ON MULTIMEDIA 3
Fig. 2: The proposed web-supervised image categorization and sub-categorization system. Backend and frontend work togetheras a pipeline for automatically collecting the labels of subcategories and associated images from the web, then selectingrepresentative images and training classifiers for image categorization and sub-categorization.
subcategories. All the images in our work will be assigned twolabels including coarse category and refinement subcategory.So their motivations and formulations are inherently differentfrom our work.
Our work is more related to the recent image sub-categorization works [31], [45], [48], which assume severalsubcategories exist in each category. However, the subcate-gories discovered by [31], [48] still only have the label of thecategory. Refinement labels of subcategories are still unavail-able. Ristin et al. [45] adopt the framework of Random Forestsand proposed a regularized objective function that takes intoaccount relations between categories and subcategories toimprove the classification accuracy. Unlike previous works,method [45] can classify images into subcategories, but onlythe NOUN subcategories. The reason is that method [45] relieson expert knowledge WordNet to obtain the semantic refine-ment subcategories. In our work, we eliminate the dependencyon expert knowledge and propose to exploit general corpusinformation to obtain the semantic refinement subcategories.The advantage of our proposed approach is that our methodcan not only classify images into NOUN subcategories, butalso into VERB, ADJECTIVE and ADVERB subcategories.
As our approach relies on weakly labeled web images, it isloosely related to the multi-instance learning works [7], [10],[40], [41]. Method [10] and [40] proposed to partition theweakly labeled web images into a set of clusters and eachcluster is treated as a “bag”, the images therein as “instances”.Correspondingly, different multi-instance learning methodswere proposed in [10] and [40]. In [7], Andrews et al. adopt aheuristic way to iteratively train the image classifier and theninfer the category labels of these instance images. Method [41]
proposed two convex optimization methods which maximizethe margin of concepts via key instance generation at theinstance-level and bag-level for locating the regions of interest(ROI) in the images automatically.
Our work is largely inspired by the following work. Aweakly supervised domain robust visual recognition systemwas recently proposed in [52] and achieved impressive per-formance for video event recognition. Niu et al. [52] firstapplied the latent domain discovering method [30] to find allthe latent domains from the training data. Then the multi-instance learning method was leveraged to cope with noise inthe labels of web training images. The main difference fromours is the formation process of the “bags”. Method [52] takesthe latent domains as “bags” while our method applies theselected subcategories. Compared to [52], our selected “bags”which derived from the general corpus have strong supervisoryinformation. This supervisory information can help us to maxi-mize the inter-class variation and simultaneously minimize theintra-class variation. In addition, as some of the previous works[40], [41], only category label can be assigned to images in[52] while our method can not only assign the category label,but also the refinement labels of subcategories.
III. WEB-SUPERVISED IMAGE CATEGORIZATION ANDSUB-CATEGORIZATION
We seek to automate the process of classifying imagesinto categories and subcategories by exploiting general corpusinformation and web images. As shown in Fig. 2, thereis a backend subsystem (classifier building) and a frontendsubsystem (categories and subcategories classifying). For thebackend system, the input is a category label that we would
IEEE TRANSACTIONS ON MULTIMEDIA 4
like to build a classification model for. Then a set of se-mantically rich subcategories is obtained by searching inGBNC, from which the noisy subcategories are removed. Afterobtaining the candidate images by retrieving the labels ofselected subcategories with image search engine, we treat eachselected subcategory as a “bag” and the images therein as“instances”. Particularly, we formulate image selecting andclassifier learning as a multi-class MIL problem by selectinga subset of images from each bag to learn the classifiers. Thefinal outputs of the backend system are (1) a classifier (e.g.,“dog”) representing the category; (2) a set of classifiers (e.g,“golden dog”, “fluffy dog” and “Eskimo dog”) correspondingto subcategories. The input of frontend is a set of images whichwill be given two labels including category and semanticrefinement subcategories.
A. Subcategories Discovering
Inspired by recent works [16], [20], [51], we can useGoogle Books Ngram English 2012 Corpus to discover thelabels of semantic refinement subcategories for modifyingthe given category. Our motivation is to find not only thesemantically rich NOUN subcategories, but also VERB, AD-JECTIVE and ADVERB subcategories. Compared to the ex-pert knowledge WordNet and ConceptNet which only haveNOUN subcategories, ngram data is much more general andexhaustive. Following [26] (see section 4.3), we specificallyuse the dependency gram data with parts-of-speech (POS) forrefinement subcategories discovering. For example, given aparent category and its corresponding POS tag (e.g., ‘jumping,VERB’), we find all its occurrences annotated with POS tagwithin the dependency gram data. Of all the ngram dependen-cies retrieved for the given category, we choose those whosemodifiers are tagged as NOUN, VERB, ADJECTIVE and AD-VERB as the candidate subcategories. We utilize these seman-tic refinement subcategories (corresponding images) to reflectthe different visual distributions of the category. The detailedsubcategories discovered in this step can be found on websitehttps://1drv.ms/f/s!Ahpq3qSTtg8NsxyjGslE2kjGcvTV.
B. Noisy Subcategories Removing
Not all the discovered subcategories are useful, some noisemay also be included (e.g., the bold subcategories in TableI). Using these noisy subcategories to retrieve images forthe category will have a negative effect on the accuracy androbustness of the classifier. To this end, we first removethese noisy subcategories before we select images and trainclassifiers for the category and subcategories. We retrieve thetop K images from image search engine for each candidatesubcategory to represent their visual distributions. By ana-lyzing the text semantics and visual distributions presentedby these subcategories, we choose the following features toseparate the useful subcategories from noise.
1) Feature selecting: From the visual relevance perspective,we want to eliminate visually less relevant subcategories (e.g.,“wood horse”, “paper boat”). The intuition is that relevantsubcategories should have a relatively small inter-visual dis-tance to its parent category and other relevant subcategories.
TABLE I: Examples of the candidate subcategories discoveredby our approach.
Cat{tiger cat, brown cat, hissing cat}{fat cat, desert cat, ginger cat}{lucky cat, tom cat, cat machine, missing cat}
We denote each image as xi and the compound featureφk = 1
k
∑ki=1 xi of K images in each subcategory to represent
visual distribution of this subcategory. Suppose a parent cate-gory Ci has N subcategories, then we will have N×(N−1)/2inter-visual distances between subcategories. We calculate theminimum, maximum, average and standard deviation of theinter-visual distances between subcategories. Besides, we alsocalculate the inter-visual distance between subcategory andits parent category. Particularly, we denote these inter-visualdistances by D = {dn, 0 6 n < N×(N−1)
2 +N} and normalizethese distances to a number in [0,1] by:
dn′
=dn −min{dn}
max{dn} −min{dn}. (1)
From the visual consistency perspective, we want to keepvisually salient and eliminate non-salient subcategories (e.g.,“missing cat”, “betting horse”). The intuition is that visuallysalient subcategories have small intra-visual distance and ex-hibit predictable visual patterns. For the K images in eachsubcategory, we calculate the K × (K − 1)/2 intra-visualdistance among these K images. We obtain the minimal, max-imal, average and standard deviation of intra-visual distancelike how we generate the inter-visual distance. Similarly, wenormalize this intra-visual distances.
From the semantic relevance perspective, we want to removesemantically less relevant subcategories (e.g., “tang horse”,“metal dog”). The intuition is that relevant subcategories tendto have a relatively small semantic distance to the parentcategory. Normalized Google Distance (NGD) constructs amethod to extract semantic similarity distance from the WorldWide Web (WWW) using Google page counts [13]. For asearch term x and search term y, NGD is defined by:
NGD(x, y) =max{log f(x), log f(y)} − log f(x, y)
logN −min{log f(x), log f(y)}(2)
IEEE TRANSACTIONS ON MULTIMEDIA 5
where f(x) denotes the number of pages containing x, f(x, y)denotes the number of pages containing both x and y and Nis the total number of web pages searched by Google.
In general, we derive the following features to separate theuseful subcategories from noise:• Normalized Google Distance between subcategory and its
parent category• Normalized visual distance between subcategory and its
parent category• Normalized minimum, maximum, average and standard
deviation of the inter-visual distance between subcategory andother subcategories• Normalized minimum, maximum, average and standard
deviation of the intra-visual distance between different imagesin subcategory
2) Classifier learning: After deriving the selected features,we train a classifier to determine whether or not a subcategoryshould be selected. To train this classifier, we label a set ofsubcategories and this labeling work only needs to be doneonce for all categories. This classifier can be, for example,Support Vector Machine (SVM) based (which is also the onewe used in our paper), Decision Tree based, etc. AlthoughSVM is not the prevailing state-of-the-art method for classifi-cation, we find our approach to be effective in pruning noisysubcategories.
C. Multi-class MIL Learning
Although the Google image search engine has ranked thereturned images, some noisy images may still be included. Inaddition, a few noisy subcategories which are not filtered outmay also induce some noise. To this end, we propose our web-supervised multi-class MIL model for noisy images removingand domain robust classifiers learning.
For ease of presentation, we denote each instance as xiwith its label yi and each bag Gm with the label Ym. Amatrix/vector is denoted by a uppercase/lowercase letter inboldface and the element-wise product between two matricesis represented by �. We also define the identity matrix asI and 0, 1 ∈ <n denote the column vectors of all zerosand ones, respectively. The transpose of a vector or matrixis represented by >. The inequality u = [u1, u2...un]
> ≥ 0means that ui ≥ 0 for i = 1,...,n. The indicator function isrepresented as δ (i = j), where δ (i = j) = 1 if i = j, andδ (i = j) = 0, otherwise.
1) Formulation: Since the retrieved web images may con-tain noise, we need to remove noise and select appropriatesamples to train robust classifiers. To this end, a binaryindicator hi ∈ {0, 1} is used to indicate whether or not traininginstance xi is selected. To be exact, hi = 1 when xi isselected, and hi = 0 otherwise. Since the precision of imagesreturned from the Google image search engine tends to havea relatively high accuracy, we define each positive bag as atleast having a portion of η positive instances. The value ofη can be estimated from some prior knowledge [40], [51].We define h = [h1, ...hN ]
> as the indicator vector, and useH = {h|
∑i∈Im hi = η |Gm| ,∀m} to represent the feasible
set of h, where Im represents the set of instance indices in
Gm, and |Gm| denotes the cardinality of Gm. We assume thereare N retrieved web images coming from C categories and Ssubcategories. zi,s ∈ {0, 1} is a binary indicator variable andtakes the value of 1 when xi belongs to the s-th subcategory,and 0 otherwise. We denote Ns =
∑Ni=1 zi,s as the number
of web training images from the s-th subcategories. Basedon MIL [40], we propose our multi-class MIL formulation asfollows:
minh,wc,s,ξm
1
2
C∑c=1
S∑s=1
‖wc,s‖2 + C1
M∑m=1
ξm (3)
s.t.1
|Gm|∑i∈Im
hi(
S∑s=1
Pi,s(wYm,s)>φ(xi)−
(wc,s)>φ(xi)) > η − ξm,∀m, s, c 6= Ym
ξm > 0,∀m
(4)
where C1 is a tradeoff parameter, ξm are slack variablesand φ(·) is the feature mapping function. Pi,s is the prob-ability that the i-th training sample comes from the s-th subcategories. It can be obtained by calculate Pi,s =
(zi,s/Ns)/∑Ss=1(zi,s/Ns). The explanation for constraint (4)
is that we force the total decision value (obtained by using theclassifier for its own category) to be larger than those obtainedby using the classifier for other categories. The motivation iswe want to reduce the bag-level loss by removing the noiseand identifying the good instances within training bags.
2) Solution: Problem (3) is a non-convex mixed integerproblem and is hard to solve directly. Inspired by recent works[40], [41], [38], we can relax the dual form of (3) as a multiplekernel learning (MKL) problem which is much easier to solve.The derivations of (3) to its below dual form is provided inthe Appendix A:
minh
maxα−1
2α>Qhα + ζ>α
s.t.∑c,s
αm,c,s = C1, ∀m,
αm,c,s > 0, ∀m, c, s.
(5)
D = M · C · S and α ∈ RD is a vector containing dualvariables αm,c,s. ζ ∈ RD is a vector, in which ζm,c,s = 0if c = Ym and ζm,c,s = η otherwise. Each elementin matrix Qh ∈ RD×D can be calculated through: Qh =(1/ |Gm| |Gm|)
∑i∈Im
∑j∈Im hihjø(xi)>ø(xj)λ(i, j, c, c, s, s).
Problem (5) is a mixed integer programming problem andis hard to directly optimize the indicator vector h. Inspired byrecent work [52], we can find the coefficients of hth
>t . For
consistent presentation, we denote d = [d1, ...dT ]>, T = |H|and the feasible set of α and d as ν and D = {d|d>1 =1,d > 0}, respectively. Then we can get the followingoptimization problem:
mind∈D
maxα∈ν
− 1
2
T∑t=1
dtα>Qhtα + ζ>α. (6)
IEEE TRANSACTIONS ON MULTIMEDIA 6
When we set the base kernel as Qht , the above problem issimilar to the MKL dual form and we are able to solve it onits primal form, which is a convex optimization problem:
mind∈D,wt,ξm
1
2
T∑t=1
‖wt‖2
dt+ C1
M∑m=1
ξm (7)
s.t.
T∑t=1
w>t ϕ(ht, Gm, c, s) > ζm,c,s − ξm, ∀m, c, s (8)
where ϕ(ht, Gm, c, s) is the feature mapping function inducedby Qht . The derivations of (6) is the dual form of (7) areprovided in the Appendix B. In the next, we will give asolution to (7).
We solve the convex problem in (7) by updating d and{wt, ξm} in an alternative way.• Update d: We firstly fix {wt, ξm} to solve d. By introduc-
ing a dual variable β for constraint d>1 = 1, the Lagrangianform of (7) can be derived as:
£ =1
2
T∑t=1
‖wt‖2
dt+ C1
M∑m=1
ξm −∑m,c,s
αm,c,s
(
T∑t=1
w>t ϕ(ht, Gm, c, s)− ζm,c,s + ξm) + β(
T∑t=1
dt − 1).
(9)Through set the derivative of (9) with respect to dt as zero,
we can get:
dt =‖wt‖√
2β, ∀t = 1, ..., T. (10)
For parameter β, ‖wt‖/√
2β is monotonically decreasing. Inaddition, parameter dt satisfy
∑Tt=1 dt = 1. Therefore, we can
use binary search method to solve β and recover dt accordingto (10).• Update wt: When d is fixed, wt can be obtained by
solving α in (6). Problem (6) is a quadratic programmingproblem w.r.t α. Since there are M · C · S variables in ourproblem, it is time-consuming to employ the existing quadraticprogramming solvers. Inspired by recent works [29], [42], weapply the cutting-plane algorithm [37] to solve this quadraticprogramming problem.
We start from a small number of base kernels and at eachiteration we add a new violating base kernel. Therefore, onlya small set of h need to be solved at each iteration and thewhole problem can be optimized more effectively. By settingthe derivatives of (9) with respect to {wt, ξt, dt} as zeros, (6)can be rewritten as:
maxβ,α∈ν
−β + ζ>α
s.t.1
2α>Qhtα 6 β, ∀t.
(11)
We solve (11) by solving α with only one constraintat the first, then add a new violating constraint iteratively.Particularly, since each constraint is associated with an ht ,we can obtain the most violated constraint by optimizing:
maxh
1
2α>Qhα (12)
Algorithm 1 Cutting-plane algorithm for solving the proposedweb-supervised multi-class MIL model.
Input:Auto-labelled image bags {(Gm, Ym)|Mm=1}, initializeyi = 1 for all xi in selected bags Gm.
1: Set t = 1 and C = {h1};2: Repeat3: t = t+ 1;4: Compute MKL to solve d and α in (6) based on C;5: //Find the most violating ht6: for each bag Gm7: Fix the labelling of instances in all other bags;8: Enumerate the candidates of yi in Gm;9: Find the optimal ym by maximizing (13);
10: end11: repeat lines 6-10 until there is no change in h;12: Add the most violating ht to the violation set C =13: C ∪ ht;14: Until The objective of (6) converges.Output:
The learnt image classifier f (x).
After a simple derivation, we can rewrite (12) as:
maxh
h>(1
2Q� (αα>))h (13)
where αi = 1/ |Gm|∑c,s αm,c,s for i ∈ Im and Q =∑
c,c,s,s φ(xi)>φ(xj)λ(i, j, c, c, s, s). Problem (13) can be
solved approximately through enumerate the binary indicatorvector h in a bag by bag fashion iteratively to maximize(13) until there is no change in h. The detailed solutions ofcut-plane algorithm for our web-supervised multi-class multi-instance learning model are described in Algorithm 1.
Since the visual distributions of the training samples fromsame category or subcategory are generally more similar thandifferent categories and subcategories, we train one classifierfor each category and each subcategory. In general, a totalof C × S classifiers fc,s(x)|c = 1, ...C, s = 1, ...S will belearned. fc,s(x) = (wc,s)
>ø(x) (for better representation, we
omit the bias term here) represents the classifier of the s-thsubcategory and the c-th category. The decision function forcategory C is obtained by integrating the learned classifiersfrom multiple subcategories: fc(xi) =
∑Ss=1 Pi,sfc,s(xi).
Given a testing image x, we want to find the labels ofthe most matched subcategory and category, whose classifierachieves the largest decision value from all the subcategoriesand categories respectively. Thus, the subcategory label ofimage x can be predicted by:
arg maxs
w>c,sφ(x) (14)
and the category label by:
arg maxc
(maxs
w>c,sφ(x)). (15)
In summary, to suppress the search error induced noisyimages, we propose a multi-class MIL model to select a subsetof training images from selected bags and simultaneouslylearning the optimal classifiers based on this selected images.
IEEE TRANSACTIONS ON MULTIMEDIA 7
IV. EXPERIMENTS
In this section, we first conduct experiments on both imagecategorization and sub-categorization to demonstrate the supe-riority of our proposed approach. Then we quantitative analysethe role of different steps contributing to the final results. Inaddition, we also analyse the parameter sensitivity and timecomplexity of our proposed approach in this section.
A. Image Categorization
The goal of this experiment is to compare the imagecategorization ability of our proposed approach with otherrelated works.
1) Experimental setting: We follow the setting in [9] andexploit web images as the training set, human-labelled imagesas the testing set. Particularly, we evaluate the performance onthe following dataset:• PASCAL VOC 2007 [23]. The PASCAL VOC 2007
dataset contains 9963 images in 20 categories. Each categoryhas training/validation data and test data. For this experiment,we only use the test data in PASCAL VOC 2007 as thebenchmark testing set. The detailed number of images for eachcategory in this experiment is shown in Table II.• STL-10 [14]. The STL-10 dataset has ten categories,
and each category of which contains 500 training images and800 test images. All of the images in STL-10 are color 96× 96 pixels. We also use the test images in STL-10 as thebenchmark testing set.• CIFAR-10 [39]. The CIFAR-10 dataset consists of 60000
32×32 images in 10 categories, with 6000 images per cat-egory, of which 5000 are training images and 1000 are testimages. Similarly, we only use the test images in CIFAR-10as the benchmark testing set.
For each category, we first obtain the semantic refine-ment subcategories by searching in the Google Books NgramCorpus. Then we retrieve the top 100 images from theGoogle image search engine for each candidate subcate-gory to represent its visual distribution. Particularly, wehave released the discovered candidate subcategories andretrieved images for all the candidate subcategories on websitehttps://1drv.ms/f/s!Ahpq3qSTtg8NsxyjGslE2kjGcvTV.
We calculate the Normalized Google Distance between sub-category and its parent category. We obtain the center of eachsubcategory by using the K-means clustering algorithm [36](k=1), then we calculate the normalized visual distance (Eu-clidean distance) between subcategory and its parent category;calculate the normalized minimum, maximum, average andstandard deviation of inter visual distance between subcategoryand other subcategories; calculate the normalized minimum,maximum, average and standard deviation of intra-visual dis-tance between different images in subcategory. We label a setof 500 positive samples and 500 negative samples to learn thelinear SVM classifier for removing noisy subcategories andselecting useful subcategories.
After we obtain the selected subcategories, the first M =50 images were selected for constructing the positive bagswhich corresponding to the selected subcategories. Negativebags can be obtained by randomly sampling a few irrelevant
TABLE II: The detailed number of images used for catego-rization in PASCAL VOC 2007.
Category Number Category NumberAero 204 tabl 190
Bicycle 239 Dog 418Bird 282 Horse 274Boat 172 Mbike 222
images. By treating each selected subcategory as a “bag” andthe images therein as “instances”, we formulate a multi-classMIL method to select a subset of training images from eachbag and simultaneously learning the optimal classifiers basedon the selected images. Particularly, we define each positivebag as having at least a portion of η = 0.7 positive instancesand set the tradeoff parameter C1 = 10−1. We will discuss theparameter setting more details in Section IV-D. To comparewith other baseline methods, we evenly select 500 images frompositive bags for each category as the training set. For thisexperiment, the feature is dense HOG [11].
2) Baselines: In order to quantify the performance ofour proposed approach, four set of weakly supervised orweb-supervised baselines are selected to compare with ourproposed approach:• SVM method. The SVM method includes multi-class
SVM [50]. For the multi-class SVM, the 500 training imagesfor each category are directly retrieved from the image searchengine with the category label.• MIL methods. The MIL methods contain instance level
method mi-SVM [7] and bag level method sMIL [10]. Formethod mi-SVM, the training images are also retrieved fromthe image search engine. Particularly, we take the proposedheuristic way to iteratively select 500 images for each categoryand train the image classifier. For method sMIL, we firstlyretrieve the candidate images from the image search engine,then we partition the candidate images into a set of clusters.Each cluster is treated as a “bag” and the images thereinas “instances”. Correspondingly, we take the proposed MILmethod to select the 500 training images for each categoryand train the image classifier.• Latent domain discovering methods. The latent domain
discovering methods include two methods DLD-MDA [32]and RVD-DA[30]. For method DLD-MDA, we firstly obtainthe candidate images from the image search engine, then wetake the hierarchical clustering technique to find the feasiblelatent domains. By treating each latent domain as a “bag” andthe images therein as “instances”, we take the proposed MILmethod to select 500 images for each category and train theimage classifier. For method RVD-DA, after we obtain thecandidate images from the image search engine, we take theproposed maximum distinctiveness and maximum learnabilityto find and separate the latent domains. Similarly, we take the
IEEE TRANSACTIONS ON MULTIMEDIA 8
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
aero bicycle bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv average
proposed MIL method to select 500 images for each categoryand train the image classifier.• Sub-categorization methods. The sub-categorization meth-
ods Sub-Cate[31] and RNCMF[45] are also used to do imagecategorization. For method Sub-Cate, the candidate images areretrieved from the image search engine. Then we discover thesubcategories during these candidate images by joint clusteringand classification. We evenly select 500 images from thesesubcategories and train image classifiers. For method RNCMF,we also obtain the candidate images from the image searchengine. We take the framework of Random Forests and theproposed regularized objective function to select 500 imagesfor each category and train the image classier.
For all the baseline methods, there are some parameters tobe set in advance. All the training images for each categoryare obtained by retrieving from the Google image searchengine. For the other parameters, we adopt the same parameter
configuration as described in their original works.3) Experimental results: The experimental results are sum-
marized in Fig. 3, Table III and Table IV. From the results,we make the following observations:
During the 20 categories in PASCAL VOC 2007, weachieved the best results in 19 categories. In the 10 categoriesof STL-10 and CIFAR-10, we obtained the best results in 7categories respectively. In addition, our approach also achievedthe best average results on all three datasets.
We observe that the MIL learning methods [10], [7],the latent domain discovering methods [32], [30], the sub-categorization methods [31], [45] and our method are generallybetter than SVM method [50]. One possible explanation isthat additional information like “bags”, latent domains orsubcategories are beneficial in web training images for imagecategorization. In specific, MIL learning methods sMIL andmi-SVM achieve better results than SVM method multi-
IEEE TRANSACTIONS ON MULTIMEDIA 9
1 0 2 0 3 0 4 0 5 0 6 0 7 05
1 0
1 5
2 0
2 5
3 0
3 5 m u l t i - S V M R N C M F S u b - C a t e M M D L O u r s
T r a i n i n g n u m b e r s
Classi
ficatio
n accu
racy
Fig. 4: Sub-categorization accuracy (%) of the different meth-ods using 50 testing images and a varying number of webtraining images for per subcategory.
SVM on all three datasets. The explanation is perhaps thatit is necessary to remove noisy images from the training setduring the process of classifier learning. Learning directlyfrom the web images without noise removing may affect theperformance of the classifier due to the presence of noisyimages in the training data.
Sub-categorization methods RNCMF and Sub-Cate gener-ally perform better than other two latent domains discoveringmethods DLD-MDA, RVD-DA and other MIL, SVM baselinemethods. One possible explanation is that classifiers learnedfrom Sub-categorization methods which exploiting subcate-gories to learn integrated classifiers are more domain robustthan MIL methods, domain discovering methods and SVMbaseline methods for image categorization.
It is interesting to observe that all classifiers have a relativelypoor performance on dataset CIFAR-10 and STL-10 than ondataset PASCAL VOC 2007. The explanation is perhaps thatall images in CIFAR-10 are cut to 32×32 and in STL-10 are96×96. Objects in these small images are placed in the middleof the image. However, our web training images and the testingimages in PASCAL VOC 2007 are both full size and containrelatively more additional objects or scenes in images.
Finally, our proposed approach achieves the best averageperformance on all three datasets, which demonstrate thesuperiority of our approach. The reason is our method simul-taneously uses the MIL technique for handling label noise inthe web training images and exploits multiple subcategoriesto learn integrated classifiers. Compared to SVM method, ourmethod not only removes the noise, but also utilizes subcate-gories to learn integrated domain robust classifiers. Comparedto MIL, latent domain discovering and sub-categorizationmethods, the multiple subcategories in our method havestrong supervisory information which was obtained from theperspective of text semantics (e.g., subcategories discoveringand noisy subcategories removing). This strong supervisoryinformation can help us to maximize the inter-class variationand simultaneously minimize the intra-class variation.
1 0 2 0 3 0 4 0 5 0 6 0 7 05
1 0
1 5
2 0
2 5
3 0
3 5
4 0 m u l t i - S V M R N C M F S u b - C a t e M M D L O u r s
T e s t i n g n u m b e r s
Classi
ficatio
n accu
racy
Fig. 5: Sub-categorization accuracy (%) of the different meth-ods using 50 web training images and a varying number oftesting images for per subcategory.
B. Image Sub-categorization
The objective of this experiment is to compare the imagesub-categorization ability of our method with four weaklysupervised or web-supervised baseline methods.
1) Experimental setting: For image sub-categorization, wechoose a subset of ImageNet as the benchmark dataset for test-ing different methods. The reason is that ImageNet which con-structed according to the WordNet has a hierarchy structure.In particular, we select five categories including “airplane”,“bird”, “cat”, “dog” and “horse” as the parent categories andall their leaf synsets as the subcategories. We are only con-cerned with the two-tier structure and deeper structure synsetsare ignored. Thus, we obtain 5 parent categories and 97 subcat-egories. A detailed number of subcategories and correspondingimages for each category in ImageNet for this experiment canbe found in Table (V). We retrieve the top 100 images foreach subcategory from the image search engine as the commonoriginal training images. So we have a total of 9700 trainingimages for 5 parent categories and 97 subcategories. For afair comparison with other baseline methods, we replace thesubcategories discovering and noisy subcategories removingprocedures with the given parent categories and subcategoriesfrom ImageNet in our work. So the initial value of C andM in our work is 5 and 97 respectively. For this experiment,the feature is 1000-dimensional bag-of-visual-words (BoW)based on densely sampled SIFT features [43]. The detailedlist of the 97 subcategories in ImageNet and the commonoriginal 9700 web training images can be found on websitehttps://1drv.ms/f/s!Ahpq3qSTtg8NsxyjGslE2kjGcvTV.
TABLE V: The detailed number of subcategories and imagesused for image sub-categorization in ImageNet.
Fig. 6: Image classification ability of SDNSR, SDMML and ours on PASCAL VOC 2007 dataset: (a) “airplane”, (b) “bird”,(c) “dog”, (d) “horse”.
2) Baselines: We compare the image sub-categorizationability of our method with four baseline methods:• multi-SVM [50]. For multi-SVM method, the class num-
ber is 97. We directly use the retrieved images from the imagesearch engine as the positive samples to learn classifiers.• Sub-Cate [31]. Method Sub-Cate takes joint clustering
and classification for subcategories discovering. For this ex-periment, the latent cluster number for each parent categoryis known and equal to the number of given subcategories.• RNCMF [45]. For RNCMF method, the labeled training
data is unavailable for both “coarse” (parent) categories and“fine” (sub) categories. The training images are retrieved fromthe image search engine which may include noise due to theerror index of image search engine. We assume there are fivetrees corresponding to our five parent categories and start therecursively learning. The depth of the tree for this experimentis all limited to two levels.• MMDL [48]. MMDL formulate image selection as a
multi-instance learning problem. For this experiment, thesubcategories are assumed as “bags” and the retrieved imagestherein as instances. We take the proposed multi-instancelearning function to select images from the retrieved imagesand learn the image classifiers.
3) Experimental results: Fig. 4 and Fig. 5 present the imagesub-categorization results achieved by different methods whenusing a varying number of training images and testing images.The accuracy is measured by the average classification rate persubcategory.
By observing Fig. 4, the best performance is achieved byour method, which produces significant improvements overmethod Sub-Cate and multi-SVM, particularly the number oftraining images over 20 for each subcategory. The reason isour method considers the noise during the process of classifierlearning. Due to the error index of the image search engine,some noise may be included. We need to select useful imagesfrom the retrieved candidate images to learn robust classifiersfor each subcategory.
From Fig. 4, we notice that the performance of the multi-SVM and Sub-Cate peaks at the value of training numbers20 or 30 and decreases monotonically after this peaks. Onepossible explanation is that the image search engine provides
images based on the estimated relevancy with respect to thequery. Images far down in the ranking list are more likely to benoise, which may result in degrading of the sub-categorizationaccuracy especially for non-robust methods multi-SVM andSub-Cate.
It is interesting to note in Fig. 4, while method RNCMF im-plements a form of noisy images removing, the classificationaccuracy did not improve with the number of training imagesincrease. One possible explanation is that the noise in thetraining data is not the only factor that affects the classificationaccuracy. The visual distribution of the selected images isanother important factor. Furthermore, the poor accuracy ofSub-Cate suggests that naively adding the number of trainingimages without considering the visual distributions not onlydoes not help but actually worsens the classification accuracy.
By observing Fig. 4 and Fig. 5, our approach comparesvery favorably with competing algorithms, in terms of differentnumbers of training and testing images. Compared to methodmulti-SVM, Sub-Cate and RNCMF, our approach achievessignificant improvements in the sub-categorization accuracy.The reason is our approach not only considers the possiblepresence of noise in the web training data, but also triesto ensure the diversity of the selected images for classifierlearning. Besides, our approach performs better than methodMMDL. The reason is we formulate image selection andclassifier learning as a multi-class MIL problem. Compared tomethod MMDL which uses the relationship between differentsubcategories for classifier learning, our method not onlyuses the relationships between various subcategories, but alsoleverage the relationships between various parent categories.Therefore, our method achieves a much better result thanMMDL.
C. Quantitative Analysis of Different Steps
Our proposed framework involves three major steps: subcat-egories discovering, noisy subcategories removing and multi-class multi-instance learning. In order to quantify the roleof different steps contributing to the final classifiers, weconstruct two new frameworks. One is based on subcategoriesdiscovering and noisy subcategories removing (which werefer to SDNSR). Another one is based on subcategories
Fig. 7: The parameter sensitiveness of C1 and η in terms ofimage classification accuracy.
discovering and multi-class multi-instance learning (which werefer to SDMML). For framework SDNSR, we first obtainthe candidate subcategories through searching in the GoogleBooks Ngram Corpus. Then we apply the noisy subcategoriesremoving procedure to get the selected subcategories. Wedirectly retrieve the top images from the image search enginefor selected subcategories to train image classifiers (withoutnoisy images removing). For framework SDMML, we alsoobtain the candidate subcategories by searching in the GoogleBooks Ngram Corpus. Then we retrieve the top images fromthe image search engine for all the candidate subcategories(without noisy subcategories removing). We apply the multi-class MIL model to select useful images and train imageclassifiers.
We compare the image classification ability of these twonew frameworks with our proposed framework. Specifically,“airplane”, “dog”, “horse” and “bird” are selected as four tar-get categories to compare the image categorization ability. Wesequentially collect [200,400,600,800,1000] images for eachcategory as the positive training samples and use 1000 fixedirrelevant negative samples to learn image classifiers. We testthe image classification ability of these three frameworks onthe PASCAL VOC 2007 dataset. The results are shown in Fig.6. By observing Fig. 6, we have the following observations:
Framework SDNSR usually performs better than SDMMLwhen the number of training images for each category is below600. One possible explanation is that the first few imagesretrieved from the image search engine tend to have a relativelyhigh accuracy. When the number of training images is below600, the noisy images induced by noisy subcategories are moreserious than those caused by the image search engine. Withthe increase of image numbers for each category, the imagesretrieved from the image search engine contain more and morenoise. In this condition, the noisy images caused by the imagesearch engine have a worse effect than those induced by noisysubcategories.
Our proposed framework outperforms both SDNSR andSDMML. This is because our approach, which takes a com-bination of noisy subcategories removing and noisy imagesfiltering, can effectively remove the noisy images induced byboth noisy subcategories and the error index of image searchengine. Our framework can maximize the filtering of noisy
3 0 0 0 4 0 0 0 5 0 0 0 6 0 0 00
2 0 0 0
4 0 0 0
6 0 0 0
8 0 0 0
1 0 0 0 0
1 2 0 0 0
1 4 0 0 0
1 6 0 0 0
O u r s
T r a i n i n g n u m b e r s
Train
ing tim
es (s
)
(a)
3 0 0 0 4 0 0 0 5 0 0 0 6 0 0 00 . 1
0 . 2
0 . 3
0 . 4
0 . 5
0 . 6
O u r s
T r a i n i n g n u m b e r s
Clas
sifica
tion a
ccur
acy
(b)
Fig. 8: The training time and image classification accuracieswith respect to various numbers of training images.
images while maintaining the diversity of the selected images,thereby reducing the negative impact of noisy images on theclassifier.
D. Parameter Sensitivity Analysis
Our proposed multi-class multi-instance learning formula-tion contains two parameters C1 and η. PASCAL VOC 2007was selected as the benchmark testing dataset to evaluate theperformance variation of our proposed approach. In particular,we vary one parameter by fixing another parameter as thedefault value. Fig. 7 presents the parameter sensitiveness ofC1 and η in terms of image classification accuracy.
By observing Fig. 7, we found our method is robust to theparameter C1 when it is varied in a certain range. Besides,the performance of our method is growing when η increasebut less than 0.7. The reason is perhaps that our training dataderived from image search engine. Due to the error index ofimage search engine, there may be too much noise in eachbag which will result in decreasing the classification accuracywhen η 6 0.7. When η increases over 0.7, the performanceof our method decreases. One possible explanation is thatthe training set is less diverse. With the increasing of η, thenumber of subcategories is decreasing, which may lead to thedegradation of domain robustness of the classifier.
E. Time Complexity Analysis
During the process of multi-class multi-instance learning,we solve the convex problem in (7) by the cutting-planealgorithm. Through finding the most violating candidate ht andsolve the MKL subproblem at each iteration, the time com-plexity of (7) can be approximately computed as T ·O(MKL),where the T is the number of iterations and the O(MKL) is thetime complexity of the MKL sub-problem. According to [17],[52], the time complexity of MKL is between t·O(LCM) andt · O((LCM)2.3), where M,L,C are the numbers of latentdomains, bags and categories respectively. t is the number ofiterations in MKL.
We take STL-10 as the testing set to evaluate our method.Particularly, we use various numbers of training images foreach category to learn the classifiers. STL-10 has 10 categoriesand we use n training images for each category, so we have a
IEEE TRANSACTIONS ON MULTIMEDIA 12
total of 10 n training images. Fig. 8 shows the training timeand image classification accuracies with respect to variousnumbers of training images. From Fig. 8, we can observethat both of training time and image classification accuracyincrease with the number of training images grows. We reportthe configuration of our experiment. Two HP PCs (3.2GHzCPU with 8 GByte RAM) were used for the web imagescollection. All the data processing and experiments wereperformed on an Acer PC (3.5GHz CPU, 16 GByte RAMand 4 GByte VRAM) with LIBSVM [49].
V. CONCLUSION
In this paper, we presented a new framework for classifyingimages into categories and subcategories. Three successivemodules were employed in the framework including subcat-egories discovering, noisy subcategories removing and multi-class multi-instance learning. Compared to existing methods,our proposed approach can not only classify images intoNOUN subcategories, but also into VERB, ADJECTIVE andADVERB subcategories. Our approach has a better semanticrefinement descriptions for the categories. To verify the effec-tiveness of our proposed approach, we conducted experimentson both image categorization and sub-categorization tasks.The experimental results demonstrated the superiority of ourproposed approach over existing weakly supervised and web-supervised approaches.
APPENDIX ATHE DERIVATIONS OF (5) IS THE DUAL FORM OF (3)
Proof: In order to deduce the dual form of (3), we introducean variable π which is defined as:
πi,c,s,s =
{Pi,s c = yi
δ(s = s) c 6= yi
where yi = Ym,∀i ∈ Im. Then, we can get:
S∑s=1
Pi,sw>yi,sφ(xi) =
S∑s=1
πi,yi,s,sw>yi,sφ(xi)
w>c,sφ(xi) =
S∑s=1
πi,c,s,sw>c,sφ(xi).
We can further rewrite the constraints in (4) as the followingforms:
1
|Gm|∑i∈Im
hi(
S∑s=1
πi,Ym,s,s(wYm,s)>φ(xi)−
S∑s=1
πi,c,s,s
(wc,s)>φ(xi)) ≥ ζm,c,s − ξm, ∀m, s, c
where ζm,c,s = 0 if c = Ym and ζm,c,s = η otherwise. Wedefine:
By introducing a dual variable αm,c,s for each constraint in(17), we can get the Lagrangian as:
£w,ξm,αm,c,s=
1
2‖w‖2 + C1
N∑i=1
ξi−∑m,c,s
αm,c,s(w>ϕ(h, Gm, c, s)− ζm,c,s + ξm).
(18)
Through set the derivatives of £ with respect to w and ξm aszeros, we can get: ∑
c,s
αm,c,s = C1 (19)
w =∑m,c,s
αm,c,sϕ(h, Gm, c, s) (20)
Through submit the obtained equalities (19) and (20) backinto (18), we can get the dual form of (3) as (5), whichcompletes the proof.
APPENDIX BTHE DERIVATIONS OF (6) IS THE DUAL FORM OF (7)
Proof: We firstly introduce a dual variable αm,c,s forconstraint in (8), then we can rewrite the Lagrangian formof (7) as:
£ =1
2
T∑t=1
‖wt‖2
dt+ C1
M∑m=1
ξm −∑m,c,s
αm,c,s
(
T∑t=1
w>t ϕ(ht, Gm, c, s)− ζm,c,s + ξm)
(21)
Through set the derivatives of £ w.r.t. wt and ξm as zerosrespectively, we can get:
wt = dt∑m,c,s
αm,c,sϕ(ht, Gm, c, s), ∀t,∑c,s
αm,c,s = C1, ∀m.(22)
Through submit (22) back into (7), we can arrive at the objectfunction in (6), which completes the proof.
IEEE TRANSACTIONS ON MULTIMEDIA 13
REFERENCES
[1] A. Mansur and Y. Kuno. Improving recognition through object sub-categorization. International Symposium on Visual Computing, 851–859,2008.
[2] Y. Yan, F. Nie, W. Li, C. Gao, Y. Yang, and D. Xu, “Image classificationby cross-media active learning with privileged information,” IEEETransactions on Multimedia, 18(12): 2494–2502, 2016.
[3] E. Chatzilari, S. Nikolopoulos, Y. Kompatsiaris, and J. Kittler, “Salic:Social active learning for image classification,” IEEE Transactions onMultimedia, 18(8): 1488–1503, 2016.
[4] S. Bai, X. Bai, and W. Liu, “Multiple stage residual model for imageclassification and vector compression,” IEEE Transactions on Multimedia,18(7): 1351–1362, 2016.
[5] J. Wang, W. Wang, R. Wang, and W. Gao, “Csps: An adaptive poolingmethod for image classification,” IEEE Transactions on Multimedia,18(6): 1000–1010, 2016.
[6] M. Jian and C. Jung, “Semi-supervised bi-dictionary learning for imageclassification with smooth representation-based label propagation,” IEEETransactions on Multimedia, 18(3): 458–473, 2016.
[7] S. Andrews, I. Tsochantaridis, and T. Hofmann. Support vector machinesfor multiple-instance learning. Advances in Neural Information Process-ing Systems, 561–568, 2002.
[8] F. R. Bach and Z. Harchaoui. Diffrac: a discriminative and flexibleframework for clustering. Advances in Neural Information ProcessingSystems, 49–56, 2008.
[9] A. Bergamo and L. Torresani. Exploiting weakly-labeled web images toimprove object classification: a domain adaptation approach. Advancesin Neural Information Processing Systems, 181–189, 2010.
[10] R. C. Bunescu and R. J. Mooney. Multiple instance learning for sparsepositive bags. International Conference on Machine Learning, 105–112,2007.
[11] N. Dalal and B. Triggs, “Histograms of oriented gradients for humandetection,” in IEEE International Conference on Computer Vision andPattern Recognition, 886–893, 2005.
[12] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan.Object detection with discriminatively trained part-based models. IEEETransactions on Pattern Analysis and Machine Intelligence, 32(9): 1627–1645, 2010.
[13] R. L. Cilibrasi and P. M. Vitanyi. The google similarity distance. IEEETransactions on Knowledge and Data Engineering, 19(3): 370–383, 2007.
[14] A. Coates, H. Lee, and A. Y. Ng. An analysis of single-layer networksin unsupervised feature learning. Ann Arbor, 1001(481): 2, 2010.
[15] K. Crammer and Y. Singer. On the algorithmic implementation ofmulticlass kernel-based vector machines. Journal of Machine LearningResearch, 2(11): 265–292, 2001.
[16] J.-B. Michel, Y. K. Shen, A. P. Aiden, A. Veres, M. K. Gray, J. P.Pickett, D. Hoiberg, D. Clancy, P. Norvig, J. Orwant, et al. Quantitativeanalysis of culture using millions of digitized books. Science, 331(6014):176–182, 2011.
[17] J. Platt, “Fast training of support vector machines using sequentialminimal optimization,” Advances in Kernel Methods, 185–208, 1999.
[18] E. Fox, E. B. Sudderth, M. I. Jordan, and A. S. Willsky. Nonparametricbayesian learning of switching linear dynamical systems. Advances inNeural Information Processing Systems, 457–464, 2009.
[19] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet:A large-scale hierarchical image database. IEEE International Conferenceon Computer Vision and Pattern Recognition, 248–255, 2009.
[20] Y. Yao, J. Zhang, F. Shen, X. Hua, J. Xu, and Z. Tang. Automatic imagedataset construction with multiple textual metadata. IEEE InternationalConference on Multimedia and Expo, 1–6, 2016.
[21] X. Hua and J. Li. “Prajna: Towards recognizing whatever you wantfrom images without image labeling,” AAAI International Conference onArtificial Intelligence, 137–144, 2015.
[22] C. Xiong, S. McCloskey, S.-H. Hsieh, and J. J. Corso. Latent domainsmodeling for visual domain adaptation. AAAI International Conferenceon Artificial Intelligence, 2860–2866, 2014.
[23] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zis-serman. The pascal visual object classes (voc) challenge. InternationalJournal of Computer Vision, 88(2): 303–338, 2010.
[24] R.-E. Fan, P.-H. Chen, and C.-J. Lin. Working set selection usingsecond order information for training support vector machines. Journalof Machine Learning Research, 6(11): 1889–1918, 2005.
[25] L.-J. Li and L. Fei-Fei. Optimol: automatic online picture collection viaincremental model learning. International Journal of Computer Vision,88(2): 147–168, 2010.
[26] Y. Lin, J.-B. Michel, E. L. Aiden, J. Orwant, W. Brockman, andS. Petrov. Syntactic annotations for the google books ngram corpus.ACL 2012 System Demonstrations, 169–174, 2012.
[27] T. L. Berg and D. A. Forsyth. Animals on the web. IEEE InternationalConference on Computer Vision and Pattern Recognition, 1463–1470,2006.
[28] R. Fergus, P. Perona, and A. Zisserman. A visual category filter forgoogle images. European Conference on Computer Vision, 242–256,2004.
[29] P. Gehler and S. Nowozin. Infinite kernel learning. Advances in NeuralInformation Processing Systems, 326–334, 2008.
[30] B. Gong, K. Grauman, and F. Sha. Reshaping visual datasets for domainadaptation. Advances in Neural Information Processing Systems, 1286–1294, 2013.
[31] M. Hoai and A. Zisserman. Discriminative sub-categorization. IEEEInternational Conference on Computer Vision and Pattern Recognition,1666–1673, 2013.
[32] J. Hoffman, B. Kulis, T. Darrell, and K. Saenko. Discovering latentdomains for multisource domain adaptation. Europe Conference onComputer Vision, 702–715, 2012.
[33] R. Fergus, L. Fei-Fei, P. Perona, and A. Zisserman. Learning objectcategories from internet image searches. Proceedings of the IEEE, 98(8):1453–1466, 2010.
[34] T. Hofmann. Probabilistic latent semantic indexing. ACM InternationalConference on Research and Development in Information Retrieval, 50–57, 1999.
[35] T. Joachims. Optimizing search engines using clickthrough data. ACMInternational Conference on Knowledge Discovery and Data Mining,133–142, 2002.
[36] D. Pelleg, A. Moore, et al., “X-means: Extending k-means with efficientestimation of the number of clusters.,” in ACM International Conferenceon Machine Learning, 102-110, 2000.
[37] J. E. Kelley, Jr. The cutting-plane method for solving convex programs.Journal of the Society for Industrial and Applied Mathematics, 8(4): 703–712, 1960.
[38] M. Kloft, U. Brefeld, S. Sonnenburg, and A. Zien. Lp-norm multiplekernel learning. Journal of Machine Learning Research, 12(3): 953–997,2011.
[39] A. Krizhevsky and G. Hinton. Learning multiple layers of features fromtiny images. 2009.
[40] W. Li, L. Duan, D. Xu, and I. W.-H. Tsang. Text-based image retrievalusing progressive multi-instance learning. IEEE International Conferenceon Computer Vision, 2049–2055, 2011.
[41] Y.-F. Li, J. T. Kwok, I. W. Tsang, and Z.-H. Zhou. A convex methodfor locating regions of interest with multi-instance learning. JointEuropean Conference on Machine Learning and Knowledge Discoveryin Databases, 15–30, 2009.
[42] Y.-F. Li, I. W. Tsang, J. T.-Y. Kwok, and Z.-H. Zhou. Tighter and convexmaximum margin clustering. In AISTATS, 344–351, 2009.
[43] T. Lindeberg. Scale invariant feature transform. Scholarpedia, 7(5): 104,2012.
[44] G. A. Miller. Wordnet: a lexical database for english. Communicationsof the ACM, 38(11): 39–41, 1995.
[45] M. Ristin, J. Gall, M. Guillaumin, and L. Van Gool. From categoriesto subcategories: large-scale image classification with partial class labelrefinement. IEEE International Conference on Computer Vision andPattern Recognition, 231–239, 2015.
[46] M. Ristin, M. Guillaumin, J. Gall, and L. Van Gool. Incremental learningof ncm forests for large-scale image classification. IEEE InternationalConference on Computer Vision and Pattern Recognition, 3654–3661,2014.
[47] R. Speer and C. Havasi. Conceptnet 5: A large semantic network forrelational knowledge. The Peoples Web Meets NLP, 161–176, 2013.
[48] X. Wang, B. Wang, X. Bai, W. Liu, and Z. Tu. Max-margin multiple-instance dictionary learning. International Conference on MachineLearning, 846–854, 2013.
[49] C. Chang and C. Lin. “Libsvm: a library for support vector machines,”ACM Transactions on Intelligent Systems and Technology, 2(3): 27, 2011.
[50] J. Weston and C. Watkins. Multi-class support vector machines.Technical Report, Citeseer, 1998.
[51] Y. Yao, X.-s. Hua, F. Shen, J. Zhang, and Z. Tang. A domain robustapproach for image dataset construction. ACM Conference on Multimedia,212–216, 2016.
[52] L. Niu, W. Li, D. Xu, and J. Cai. Visual recognition by learningfrom web data via weakly supervised domain generalization. IEEETransactions on Neural Networks and Learning Systems, 2016.
![From Categories to Subcategories: Large-Scale Image ......3.1. NCM Forests NCM forests belong to the family of random forests [9], which combine classifiers consisting of T decision](https://static.documents.pub/doc/80x56/5ec5599913b08355f20aa56e/from-categories-to-subcategories-large-scale-image-31-ncm-forests-ncm.jpg)
