Where is the bear?

Andy Rosales EliasMentor: Nevena Golubovic

Advisors: Chandra Krintz, Rich WolskiLab: RACELab

Department of Computer Science

Implementing Machine Learning-Based Image Recognition for Animal Detection

Carters New Agency http://www.catersnews.com/

Carters New Agency http://www.catersnews.com/

The Problem of Accuracy, Cost, And Efficiency

= + +

Data Reliability

The Problem of Accuracy, Cost, And Efficiency

= + +

Animal Recognition Using Different Machine Learning Frameworks

Output: “Bear” Output: “None”

Using Convolutional Neural Networks (CNN) to Recognize Animals

-Zhong, Zhuoyao et al. “High Performance Offline Handwritten Chinese Character Recognition Using GoogLeNet and Directional Feature Maps.” -"Multi-column deep neural networks for image classification". 2012 IEEE Conference on Computer Vision and Pattern -Recognition-Ciresan, Dan; Ueli Meier; Jonathan Masci; Luca M. Gambardella; Jurgen Schmidhuber. "Flexible, High Performance Convolutional Neural Networks for Image Classification"

- Almost every highly ranked team used CNN as their basic framework

- Foundation of GoogLeNet and AlexNet

Dog

Cat

How Convolutional Neural Networks Classify Images

Feature extractor black box

Deactivated neuron

Activated neuron

}

}

Input layer

Characteristic layers

Output layer

How Convolutional Neural Networks Classify Images

Feature extractor black box

Dog

Cat

Deactivated neuron

Activated neuron

}

}

Input layer

Characteristic layers

Output layer

How Convolutional Neural Networks Classify Images

Feature extractor black box

Dog

Cat

Deactivated neuron

Activated neuron

}

}

Input layer

Characteristic layers

Output layer

How Convolutional Neural Networks Classify Images

Feature extractor black box

Dog

Cat

Deactivated neuron

Activated neuron

}

}

Input layer

Characteristic layers

Output layer

Re-training an Existing CNN Model

Deactivated neuron

Activated neuron

● Use a pre-trained CNN model

Tree

Deer

Pond

Characteristic layers(pre-trained model)

Re-training an existing CNN Model

Deactivated neuron

Activated neuron

“deer”

Extra filtering layer

● Use a pre-trained CNN model● Add an extra filtering layer

Characteristic layers(pre-trained model)

Results After Classifying 12210 Images Using Pre-trained CNN Model

Gazelle 9%

Mountain bike - 10%

TensorFlow - Top 5 Categories

Barrow17%

MountainTent38%

Other - 26%

Results After Classifying 12210 Images Using TensorFlow Re-trained In Four Categories

Empty/other - 76%

Deer - 19%Coyote 3%

Bear0.5%

Re-training Dataset:● Bear: 170 images● Deer: 489 images● Coyote: 259 images● Empty/other: 21 images

Re-trained CNN: Tensorflow

Accuracy of TensorFlow: Before And After Re-Training

No Re-training Retrained in four categories

Average accuracy: 35% Average accuracy: 73%

Metadata Extraction & Fusion

● Learn more about how animals are affected by the environment.

● Analyze this data to learn more about certain species.

● Time and date can be extracted from the EXIF data● Temperature needs to be recognized using OCR

Extracting metadata from images

Recognizing temperature digits using OCRHave temperature digits recognized automatically

Recognizing temperature digits using OCRHave temperature digits recognized automatically

? ?

Training phase - labeling digitsLabel each digit in the font family in order to train the model

Training phase - labeling digits

0

Manual labeling

Label each digit in the font family in order to train the model

Training phase - labeling digits

0 1

Manual labeling

Label each digit in the font family in order to train the model

Training phase - labeling digits

0 1 2

Manual labeling

Label each digit in the font family in order to train the model

Label each digit in the font family in order to train the model

Training phase - labeling digits

0 1 2 3 4 5 6 7 8 9 -

Manual labeling

Recognizing temperature digits using OCR

● The trained model will recognize any unlabeled character● Which number is the “nearest” in similarity with the labeled data

9 4

Collecting metadata with hidden, motion-triggered cameras at the Sedgwick Reserve

Camera 1 Camera 2 Camera 3

Camera 1 Camera 2 Camera 3

Collecting metadata with hidden, motion-triggered cameras at the Sedgwick Reserve

UnknownClassifier

84

1319

How each OCR algorithm processes temperature images

Previous OCR

UnknownClassifier

84

1319

Camera 1

Camera 2

Camera 3

94

73

79

Previous OCR

New OCR

How each OCR algorithm processes temperature images

Analyzing the accuracy and processing time of both OCR algorithms

Accuracy % Time in seconds

- Re-train TensorFlow with labeled images from Sedgwick Reserve- Collect more data to build a bigger dataset- Move everything to the cloud

The Road Ahead

Manual Labeling

Improve CNN Model

AcknowledgmentsSpecial Thanks To:

Mentor: Nevena GolubovicAdvisors: Chandra Krintz and Rich WolskiProgram Coordinator: Stephanie Mendes

CSEP

Recognizing temperature digits using OCR

● We need to automatically have these numbers recognized

Recognizing temperature digits using OCR

● We need to automatically have these numbers recognized● Find the edges on the image

Recognizing temperature digits using OCR

● We need to automatically have these numbers recognized● Find the edges on the image● With the edges, we know the dimensions of each digit/symbol

Recognizing temperature digits using OCR

● We need to automatically have these numbers recognized● Find the edges on the image● With the dimensions, we ignore characters with small area (degree symbol)

● However, numbers are still unlabeled

? ?