+ All Categories
Home > Technology > Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer-wise CNN Surgery for Visual Sentiment Prediction

Date post: 12-Aug-2015
Category:

Technology
Upload: xavier-giro
View: 76 times
Download: 1 times
Download Report this document
Share this document with a friend
Popular Tags:
related work
cnn introduction
future work
train test experimental
deep networks
sentiment of images
fold crossvalidation
problem definition
68
LAYER-WISE CNN SURGERY FOR VISUAL SENTIMENT PREDICTION Víctor Campos Xavier Giró Amaia Salvador Brendan Jou July 20th 2015
Transcript
Page 1: Layer-wise CNN Surgery for Visual Sentiment Prediction

LAYER-WISE CNN SURGERY FOR VISUAL SENTIMENT

PREDICTION

Víctor Campos Xavier Giró Amaia Salvador Brendan Jou

July 20th 2015

Page 2: Layer-wise CNN Surgery for Visual Sentiment Prediction

Outline

1. Introduction2. Related work3. Methodology and results4. Conclusions5. Future work

2

Page 3: Layer-wise CNN Surgery for Visual Sentiment Prediction

3

Introduction: motivation

Page 4: Layer-wise CNN Surgery for Visual Sentiment Prediction

4

Introduction: motivation

Page 5: Layer-wise CNN Surgery for Visual Sentiment Prediction

Introduction: motivation

5

Page 6: Layer-wise CNN Surgery for Visual Sentiment Prediction

6

Introduction: problem definition▷ What? ▷ How?

Page 7: Layer-wise CNN Surgery for Visual Sentiment Prediction

▷ What? Predict the sentiment that an image provokes to a human▷ How?

7

Introduction: problem definition

Page 8: Layer-wise CNN Surgery for Visual Sentiment Prediction

▷ What? Predict the sentiment that an image provokes to a human▷ How?

8

Introduction: problem definition

Page 9: Layer-wise CNN Surgery for Visual Sentiment Prediction

▷ What? Predict the sentiment that an image provokes to a human▷ How? Using Convolutional Neural Networks (CNNs)

9

CNN

Introduction: problem definition

Page 10: Layer-wise CNN Surgery for Visual Sentiment Prediction

10

CNN

Introduction: example

Page 11: Layer-wise CNN Surgery for Visual Sentiment Prediction

11

CNN

Introduction: example

Page 12: Layer-wise CNN Surgery for Visual Sentiment Prediction

Outline

1. Introduction2. Related work3. Methodology and results4. Conclusions5. Future work

12

Page 13: Layer-wise CNN Surgery for Visual Sentiment Prediction

Related work: low-level descriptors

13

Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010, October). Analyzing and predicting sentiment of images on the social web. In Proceedings of the international conference on Multimedia (pp. 715-718). ACM.

Machajdik, J., & Hanbury, A. (2010, October). Affective image classification using features inspired by psychology and art theory. In Proceedings of the international conference on Multimedia (pp. 83-92). ACM.

Page 14: Layer-wise CNN Surgery for Visual Sentiment Prediction

14

Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S. F. (2013, October). Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM international conference on Multimedia (pp. 223-232). ACM.

Related work: SentiBank

Page 15: Layer-wise CNN Surgery for Visual Sentiment Prediction

Related work: CNNs for sentiment prediction

15

You, Q., Luo, J., Jin, H., & Yang, J. (2015). Robust image sentiment analysis using progressively trained and domain transferred deep networks. In The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI).

Page 16: Layer-wise CNN Surgery for Visual Sentiment Prediction

Outline

1. Introduction2. Related work3. Methodology and results

a. Convolutional Neural Networksb. Datasetsc. Experimental setup and results

4. Conclusions5. Future work

16

Page 17: Layer-wise CNN Surgery for Visual Sentiment Prediction

Convolutional Neural Networks

17

Krizhevsky, A.; Sutskever, I. & Hinton, G. E.: ImageNet Classification with Deep Convolutional Neural Networks. In: NIPS., 2012

Page 18: Layer-wise CNN Surgery for Visual Sentiment Prediction

Outline

1. Introduction2. Related work3. Methodology and results

a. Convolutional Neural Networksb. Datasetsc. Experimental setup and results

4. Conclusions5. Future work

18

Page 19: Layer-wise CNN Surgery for Visual Sentiment Prediction

Datasets

19

Flickr Twitter

Authors Borth et al. (2013) You et al. (2015)

Size ~500k 1269

Annotation method Textual tags5 human

annotators

Page 20: Layer-wise CNN Surgery for Visual Sentiment Prediction

Datasets

20

Size

Flickrdataset

Quality of the annotations

Twitterdataset

Page 21: Layer-wise CNN Surgery for Visual Sentiment Prediction

Datasets

21

Size

Flickrdataset

Quality of the annotations

Twitterdataset

Page 22: Layer-wise CNN Surgery for Visual Sentiment Prediction

Outline

1. Introduction2. Related work3. Methodology and results

a. Convolutional Neural Networksb. Datasetsc. Experimental setup and results

4. Conclusions5. Future work

22

Page 23: Layer-wise CNN Surgery for Visual Sentiment Prediction

Experimental setup: 5-fold cross-validation

Dataset

Page 24: Layer-wise CNN Surgery for Visual Sentiment Prediction

Experimental setup: 5-fold cross-validation

Page 25: Layer-wise CNN Surgery for Visual Sentiment Prediction

Train Test

Experimental setup: 5-fold cross-validation

Page 26: Layer-wise CNN Surgery for Visual Sentiment Prediction

Train Test

Mean ± Std. Dev.

Experimental setup: 5-fold cross-validation

Page 27: Layer-wise CNN Surgery for Visual Sentiment Prediction

27

ARCHITECTURECaffeNet

Experimental setup: CNN

Page 28: Layer-wise CNN Surgery for Visual Sentiment Prediction

28

ARCHITECTURECaffeNet

SOFTWARE[Jia’14]

Experimental setup: CNN

Page 29: Layer-wise CNN Surgery for Visual Sentiment Prediction

Experimental setup: CNN

29

Pre-trainedModel

ARCHITECTURECaffeNet

SOFTWARE[Jia’14]

Page 30: Layer-wise CNN Surgery for Visual Sentiment Prediction

Experimental setup: outline

1. Fine-tuning CaffeNet2. Layer by layer analysis3. Layer ablation4. Layer addition

30

Page 31: Layer-wise CNN Surgery for Visual Sentiment Prediction

Fine-tuning CaffeNet

31

Page 32: Layer-wise CNN Surgery for Visual Sentiment Prediction

Fine-tuning CaffeNet

32

Page 33: Layer-wise CNN Surgery for Visual Sentiment Prediction

Fine-tuning CaffeNet

33

Page 34: Layer-wise CNN Surgery for Visual Sentiment Prediction

Fine-tuning CaffeNet

34

Pre-trainedmodel

Page 35: Layer-wise CNN Surgery for Visual Sentiment Prediction

Data augmentation (oversampling)

35

CNN

Page 36: Layer-wise CNN Surgery for Visual Sentiment Prediction

Data augmentation (oversampling)

36

CNN

Page 37: Layer-wise CNN Surgery for Visual Sentiment Prediction

Data augmentation (oversampling)

37

CNN

Page 38: Layer-wise CNN Surgery for Visual Sentiment Prediction

Data augmentation (oversampling)

38

CNN

Page 39: Layer-wise CNN Surgery for Visual Sentiment Prediction

Data augmentation (oversampling)

39

CNN

Page 40: Layer-wise CNN Surgery for Visual Sentiment Prediction

Data augmentation (oversampling)

40

CNN

Page 41: Layer-wise CNN Surgery for Visual Sentiment Prediction

Data augmentation (oversampling)

41

CNN

Page 42: Layer-wise CNN Surgery for Visual Sentiment Prediction

Fine-tuning CaffeNet

42

Page 43: Layer-wise CNN Surgery for Visual Sentiment Prediction

Experimental setup: outline

1. Fine-tuning CaffeNet2. Layer by layer analysis3. Layer ablation4. Layer addition

43

Page 44: Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer by layer analysis

44

Page 45: Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer by layer analysis

45

Page 46: Layer-wise CNN Surgery for Visual Sentiment Prediction

Experimental setup: outline

1. Fine-tuning CaffeNet2. Layer by layer analysis3. Layer ablation4. Layer addition

46

Page 47: Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer ablation

47

Raw ablation

2-neuron on top

Page 48: Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer ablation

48

Page 49: Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer ablation

49

Page 50: Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer ablation

50

~16Mparams(~25%)

Page 51: Layer-wise CNN Surgery for Visual Sentiment Prediction

Experimental setup: outline

1. Fine-tuning CaffeNet2. Layer by layer analysis3. Layer ablation4. Layer addition

51

Page 52: Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer addition

52

Page 53: Layer-wise CNN Surgery for Visual Sentiment Prediction

Layer addition

53

Page 54: Layer-wise CNN Surgery for Visual Sentiment Prediction

Outline

1. Introduction2. Related work3. Methodology and results4. Conclusions5. Future work

54

Page 55: Layer-wise CNN Surgery for Visual Sentiment Prediction

Conclusions

55

Pre-trainedmodel

Page 56: Layer-wise CNN Surgery for Visual Sentiment Prediction

56

CNN

Conclusions

Page 57: Layer-wise CNN Surgery for Visual Sentiment Prediction

Conclusions

57

Page 58: Layer-wise CNN Surgery for Visual Sentiment Prediction

Outline

1. Introduction2. Related work3. Methodology and results4. Conclusions5. Future work

58

Page 59: Layer-wise CNN Surgery for Visual Sentiment Prediction

Future work

59

Size

Flickrdataset

Quality of the annotations

Twitterdataset

Page 60: Layer-wise CNN Surgery for Visual Sentiment Prediction

Future work

60

Size

Flickrdataset

Quality of the annotations

Twitterdataset

New Flickr

dataset

Page 61: Layer-wise CNN Surgery for Visual Sentiment Prediction

Experimental setup: introduction

61

Model

ARCHITECTURECaffeNet

SOFTWARE[Jia’14]

DATASET[Jou’15]

Page 62: Layer-wise CNN Surgery for Visual Sentiment Prediction

62

Page 63: Layer-wise CNN Surgery for Visual Sentiment Prediction

Acknowledgements

63

Financial supportTechnical support

Albert Gil Josep Pujal

Page 64: Layer-wise CNN Surgery for Visual Sentiment Prediction
Page 65: Layer-wise CNN Surgery for Visual Sentiment Prediction
Page 66: Layer-wise CNN Surgery for Visual Sentiment Prediction

Evaluation metric: accuracy

Page 67: Layer-wise CNN Surgery for Visual Sentiment Prediction

Top-5 scores

Page 68: Layer-wise CNN Surgery for Visual Sentiment Prediction

Receptive fields visualizationCONV5, unit 49:

CONV5, unit 51:


Recommended
sept 2007 CardMaps · 2017. 7. 20. · you are very pager-naps pagernaps vaeœl\ues sentiment sentiment sentiment sentiment se co hello sent Th ema s sentiment sentiment rñdÞS sentime

sept 2007 CardMaps · 2017. 7. 20. · you are very pager-naps pagernaps vaeœl\ues sentiment sentiment sentiment sentiment se co hello sent Th ema s sentiment sentiment rñdÞS sentime

Documents

Sentiment Analysis of Tweets by CNN utilizing Tweets with ... · emoji , or ,we manually judged sentiments for each class until 50 tweets which text and emoji express consistent sentiment

Sentiment Analysis of Tweets by CNN utilizing Tweets with ... · emoji , or ,we manually judged sentiments for each class until 50 tweets which text and emoji express consistent sentiment

Documents

01 - CNNcdn.cnn.com/cnn/CNN/Programs/airport.network/cnn-apn-media-kit-2019.pdfThe top-rated Emmy Award-winning CNN Original Series which includes ... our airports use social media

01 - CNNcdn.cnn.com/cnn/CNN/Programs/airport.network/cnn-apn-media-kit-2019.pdfThe top-rated Emmy Award-winning CNN Original Series which includes ... our airports use social media

Documents

CNN-Based Food Image Segmentation Without Pixel-Wise ... · CNN-Based Food Image Segmentation Without Pixel-Wise Annotation 453 a black box for analysis and understanding of it. Guided

CNN-Based Food Image Segmentation Without Pixel-Wise ... · CNN-Based Food Image Segmentation Without Pixel-Wise Annotation 453 a black box for analysis and understanding of it. Guided

Documents

Deep-Sentiment: Sentiment Analysis Using …Deep-Sentiment: Sentiment Analysis Using Ensemble of CNN and Bi-LSTM Models Shervin Minaee , Elham Azimi , AmirAli Abdolrashidiy New York

Deep-Sentiment: Sentiment Analysis Using …Deep-Sentiment: Sentiment Analysis Using Ensemble of CNN and Bi-LSTM Models Shervin Minaee , Elham Azimi , AmirAli Abdolrashidiy New York

Documents

Convolutional Neural Network - CNN · Introduction CNN Layers CNN Models Popular Frameworks Papers References Convolutional Neural Network - CNN Eduardo Todt, Bruno Alexandre Krinski

Convolutional Neural Network - CNN · Introduction CNN Layers CNN Models Popular Frameworks Papers References Convolutional Neural Network - CNN Eduardo Todt, Bruno Alexandre Krinski

Documents

SCA-CNN: Spatial and Channel-Wise Attention in ... · ral Network, dubbed SCA-CNN, which learns to pay at-tention to every feature entry in the multi-layer 3D feature maps. Figure

SCA-CNN: Spatial and Channel-Wise Attention in ... · ral Network, dubbed SCA-CNN, which learns to pay at-tention to every feature entry in the multi-layer 3D feature maps. Figure

Documents

Sentiment Analysis & Opinion Mining€¦ · Sentiment Analysis Sentiment Classification System Experimente Perspektiven * Abbildung dem Sinn nach entnommen aus Heyer (2006: 5). Sentiment

Sentiment Analysis & Opinion Mining€¦ · Sentiment Analysis Sentiment Classification System Experimente Perspektiven * Abbildung dem Sinn nach entnommen aus Heyer (2006: 5). Sentiment

Documents

Sentiment Analysis Using Deep Learning Technique …acadpubl.eu/jsi/2017-114-7-ICPCIT-2017/articles/11/6.pdf · Sentiment Analysis Using Deep Learning Technique CNN with KMeans ...

Sentiment Analysis Using Deep Learning Technique …acadpubl.eu/jsi/2017-114-7-ICPCIT-2017/articles/11/6.pdf · Sentiment Analysis Using Deep Learning Technique CNN with KMeans ...

Documents

for Image Captioning - arXivwoman sitting at a table CNN (VGG19) with conv5_3conv5_4 conv5_3 conv5_4 Figure 1. The illustration of channel-wise visual attention in two convolutional

for Image Captioning - arXivwoman sitting at a table CNN (VGG19) with conv5_3conv5_4 conv5_3 conv5_4 Figure 1. The illustration of channel-wise visual attention in two convolutional

Documents

Sentiment Analysis of Tweets by CNN utilizing Tweets with Emoji as … · 2: Tweets text and emoji express inconsistent sentiment. Forexample:仕事を失った!(Lostmyjob!) As shown

Sentiment Analysis of Tweets by CNN utilizing Tweets with Emoji as … · 2: Tweets text and emoji express inconsistent sentiment. Forexample:仕事を失った!(Lostmyjob!) As shown

Documents

CNN LIVE EVENT/SPECIAL - 2012 Presidential … LIVE EVENT/SPECIAL ... Tonight's debate is airing on CNN, CNN International, CNN en Espanol and the American Forces Network. We want

CNN LIVE EVENT/SPECIAL - 2012 Presidential … LIVE EVENT/SPECIAL ... Tonight's debate is airing on CNN, CNN International, CNN en Espanol and the American Forces Network. We want

Documents

Lec 4. CNN Architecturesalinlab.kaist.ac.kr/resource/Lec4_CNN_architectures.pdf · 2020-04-25 · •Linear transformation done in pixel-wise •Can be represented by a matrix •Useful

Lec 4. CNN Architecturesalinlab.kaist.ac.kr/resource/Lec4_CNN_architectures.pdf · 2020-04-25 · •Linear transformation done in pixel-wise •Can be represented by a matrix •Useful

Documents

CNN StoryLine

CNN StoryLine

News & Politics

Sentiment Analysis of Greek Tweets and Hashtags using ...hashtag.nonrelevant.net/Sentiment Analysis of Greek... · sentiment rating for the Greek tweets, for a variety of sentiment

Sentiment Analysis of Greek Tweets and Hashtags using ...hashtag.nonrelevant.net/Sentiment Analysis of Greek... · sentiment rating for the Greek tweets, for a variety of sentiment

Documents

Riding the tide of sentiment change: sentiment analysis ... · Riding the tide of sentiment change: sentiment analysis with evolving online reviews ... market intelligence, and customer

Riding the tide of sentiment change: sentiment analysis ... · Riding the tide of sentiment change: sentiment analysis with evolving online reviews ... market intelligence, and customer

Documents

UTILITIES - CNN

UTILITIES - CNN

Documents

CNN Center

CNN Center

Documents