Transfer Learning on anAutoencoder-based Deep Network

Leandro F. Ariza JiménezAdvisor: Olga Lucía Quintero, PhD.

Mathematical Modeling Research Group - GRIMMATSchool of Mathematical Sciences

Universidad EAFIT

September 6, 2016

1 / 58

Previous talk

• Big Data discussion• Clustering algorithms overview• Introduction to deep learning• Future attendance to Cornell University workshop

2 / 58

Cornell-SJTU International Workshop

3 / 58

Research question

How well does a deep network work if we perform aclassification task B

based on what the deep network learned from a previousclassification task A?

4 / 58

Project overview

• Two datasets of pictures (Digits and Leaves)• Small deep network with two hidden layers• Train each hidden layers using autoencoders• Perform 3 experiments to conclude about

5 / 58

Autoencoders

X

Z

Arquitecture

X̃

Model

z (i) = W1x(i) + b1

x̃ (i) = W2z(i) + b2

Objective Function

J (W1, b1,W2, b2) =m∑i=1

(W2

(W1x

(i) + b1

)+ b2 − x (i)

)2

6 / 58

Autoencoders

Source:[3]7 / 58

Digits datasetMNIST database of handwritten digits [4]:

• Normalized greyscale images of 28x28 pixels• Training Set (Tr. Set) with 5000 images (500 per digitapprox.)

• Test Set (Te. Set) with 5000 images (500 per digit)8 / 58

Leaves dataset

Leaves dataset taken by Markus Weber [5]:

• 186 RGB images of 896x592 pixels• 66 images of type 1• 60 images of both type 2 and type 3

9 / 58

Leaves dataset - Pre-processed

Leaves dataset taken by Markus Weber [5]:

• Normalized and inverted greyscale images of 28x28 pixels• Training Set (Tr. Set) with 120 images (40 per type)• Test Set (Te. Set) with 60 images (20 per type)

10 / 58

Proposed deep network

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

11 / 58

Exp. 1 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

12 / 58

Exp. 1 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

13 / 58

Exp. 1 - Digits dataset - Pre-training

I A I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Image(784) Softmax

14 / 58

Exp. 1 - Digits dataset - Pre-training

I A I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Image(784) Softmax

15 / 58

Exp. 1 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

16 / 58

Exp. 1 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

17 / 58

Exp. 1 - Digits dataset - Pre-training

I A I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50)

Hidden1(100)

18 / 58

Exp. 1 - Digits dataset - Pre-training

I A I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50)

Hidden1(100)

19 / 58

Exp. 1 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

20 / 58

Exp. 1 - Digits dataset - Training

I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

21 / 58

Exp. 1 - Digits dataset - Training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

22 / 58

Exp. 1 - Digits dataset - Testing

Outpu

t

Te. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

23 / 58

Exp. 1 - Digits dataset - Fine tuning

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

24 / 58

Exp. 1 - Digits dataset - Fine tuning

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

25 / 58

Exp. 1 - Digits dataset - Fine tuning

Outpu

t

Te. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

26 / 58

Exp. 1 - Digits dataset - Results

1 2 3 4 5 6 7 8 9 0

Digits

60

70

80

90

100

Accu

racy %

Experiment 1 - Results

Testing set

Fine tuning

27 / 58

Exp. 2 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

28 / 58

Exp. 2 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

29 / 58

Exp. 2 - Digits dataset - Pre-training

I A I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Image(784) Softmax

30 / 58

Exp. 2 - Digits dataset - Pre-training

I A I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Image(784) Softmax

31 / 58

Exp. 2 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

32 / 58

Exp. 2 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

33 / 58

Exp. 2 - Digits dataset - Pre-training

I A I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50)

Hidden1(100)

34 / 58

Exp. 2 - Digits dataset - Pre-training

I A I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50)

Hidden1(100)

35 / 58

Exp. 2 - Digits dataset - Pre-training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

36 / 58

Exp. 2 - Digits dataset - Training

I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

37 / 58

Exp. 2 - Digits dataset - Training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

38 / 58

Exp. 2 - Digits dataset - Testing

Outpu

t

Te. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

39 / 58

Exp. 2 - Leaves dataset - Fine tuning

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

40 / 58

Exp. 2 - Leaves dataset - Fine tuning

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

41 / 58

Exp. 2 - Digits dataset - Fine tuning

Outpu

t

Te. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

42 / 58

Exp. 2 - Leaves dataset - Results

1 2 3

Leaves type

20

40

60

80

100

Accu

racy %

Experiment 2 - Results

Test

Fine tuning

43 / 58

Exp. 3 - Leaves dataset - Transfer Learning

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

44 / 58

Exp. 3 - Leaves dataset - Transfer Learning

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

45 / 58

Exp. 3 - Leaves dataset - Training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

46 / 58

Exp. 3 - Leaves dataset - Training

I

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

47 / 58

Exp. 3 - Leaves dataset - Training

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

48 / 58

Exp. 3 - Leaves dataset - Testing

Outpu

t

Te. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

49 / 58

Exp. 3 - Leaves dataset - Fine tuning

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

50 / 58

Exp. 3 - Leaves dataset - Fine tuning

Outpu

t

Tr. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

51 / 58

Exp. 3 - Digits dataset - Fine tuning

Outpu

t

Te. Set

Image(784)

Hidden1(100)

Hidden2(50) Softmax

52 / 58

Exp. 3 - Leaves dataset - Results

1 2 3

Leave type

20

40

60

80

100

Accura

cy %

Experiment 2 & 3 - Results

Test Exp. 2

Fine tuning Exp. 2

Test Exp. 3

Fine tuning Exp. 3

53 / 58

Conclusions

• We can effectivily train a deep network by means ofautoencoders.

• A deep network can work well when we apply a transferlearning procedure.

• Fine tuning procedure can improves the overallperformance of a deep network.

• MATLAB® allows a straightforward implementation ofdeep networks.

54 / 58

Future work

• Implement more experiments, e.g. interchange roles inExperiment 3.

• Manipulate the technical specifications of eachcomponent in the deep network.

• Repeat this experiments using diferente datasets.

55 / 58

References I

[1] Taylor B. Arnold.Lecture 18 - Transfer Learning and Computer Vision I.http://euler.stat.yale.edu/~tba3/stat665/lectures/lec18/lecture18.pdf.Accesed: 24/07/2016.

[2] Li Fei-Fei, R. Fergus, and P. Perona.Learning Generative Visual Models from Few TrainingExamples: An Incremental Bayesian Approach Tested on101 Object Categories.In Conference on Computer Vision and Pattern RecognitionWorkshop, 2004. CVPRW ’04., pages 178–178, 2004.

56 / 58

References II[3] Quoc V. Le.

A Tutorial on Deep Learning, Part 2: Autoencoders,Convolutional Neural Networks and Recurrent NeuralNetworks.https://cs.stanford.edu/~quocle/tutorial2.pdf.Accesed: 24/07/2016.

[4] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998.

[5] The Computational Vision Group.Leaves dataset taken by Markus Weber (1999).http://www.vision.caltech.edu/Image_Datasets/leaves/leaves.tar.Accesed: 24/07/2016.

57 / 58

References III

[6] The MathWorks, Inc.Training a Deep Neural Network for Digit Classification.http://www.mathworks.com/help/nnet/examples/training-a-deep-neural-network-for-digit-classification.html.Accesed: 24/07/2016.

58 / 58