Damage Assessment from Social Media Imagery Data During Disasters

Dat T. Nguyen, Ferda Ofli, Muhammad Imran, Prasenjit MitraQatar Computing Research Institute, Qatar

The Pennsylvania State University, University Park, PA, USA

Partners & Clients:

New York (Suffolk) Emergency Management Dept.

Types of Information on Twitter

- Twitter data from 13 recent crises

- Over 100,000 tweets

- Information types

- Types of sources

Source: Qatar Computing Research Institute - Published in World Humanitarian Data and Trends 2014 (UN OCHA)

The Value of Timely Information During Disasters

Based on FEMA large-scale survey among emergency management professionals across the US.

Info

rmat

ion

val

ue

When information is too late

The Value of Timely Information During Disasters

Based on FEMA large-scale survey among emergency management professionals across the US.

Info

rmat

ion

val

ue

When information is too late

2013 Pakistan EarthquakeSeptember 28 at 07:34 UTC

2010 Haiti EarthquakeJanuary 12 at 21:53 UTC

Social Media Data and Opportunities

Social MediaPlatforms

Availability of Immense Data:

Around 16 thousands tweetsper minute were posted duringthe hurricane Sandy in the US.

Opportunities:

- Early warning and event detection

- Situational awareness

- Actionable information

- Rapid crisis response

- Post-disaster analysis

Disease outbreaks

“A picture is worth a thousand words.”Images from 3 Different Disasters

Time-Critical Events and Information Gaps

Info. Info. Info.

Disaster event (earthquake, flood) Destruction, Damage

Information gathering

Humanitarian organizations and local administrationNeed information to help and launch response

Information gathering, especially in real-time, is the most challenging part

Relief operations & reconstruction

Disaster

Government orgs.

Tweet4Act: Automatic Image Processing Pipeline

Presented at ASONAM’17 as demo

Damage Severity Assessment from Images

Task: Our Task is to classify each incoming imageInto one of the three classes.

Challenges

• Task complexity: lack of labeled data, ill-defined objects

• Poor signal-to-noise ration: social media data is extremely noisy. E.g., duplicates, irrelevant

• Task subjectivity: confusion between damage severity classes “severe” and “mild”

• Cold-start issue: first few hours of a disaster are critical, learning ML classifiers needs labeled data

Images Datasets: Twitter + Google

Twitter messages collected using

- Damaged building- Damaged road- Damaged bridge

Queries we used:

Human Annotations

We used AIDR (volunteers) and Crowdflower (paid workers)

The purpose of this task is to assess the severity of damage shown in an image…

1. Severe damage

Substantial destruction, a non-livableOr non-useable building, a non-crossable Bridge, a non-drivable road

2. Mild damage

Damage generally exceeding minor (e.g., 50% of a building is damaged), partial loss of amenity/roof, part of bridge is unusable or needs repairs

3. Little-to-no damage

Images that show damage-free infrastructureOr small cracks, wear and tear due to age

Three classes:

Instructions:

Human Annotations

We used AIDR (volunteers) and Crowdflower (paid workers)

Crowdflower annotations

AIDR was used during the actual event.

Learning Schemes

1. Baseline (PHOW + SVM): Pyramid Histogram of Visual Words (PHOW) featureswith linear SVM

2. Pre-trained CNN as feature extractor: We used VGG-16 network trained on the ImageNet dataset1.2M images and 1000 classes. We used fc7 layer i.e., removed the last layerto get a 4097-dimensional vector for every image.

3. Fine-tuning a pre-trained CNN: Used existing weights of a pre-trained CNN as an initialization for our datasetWhere last layer representing our task (3 classes)

Learning Settings

1. Event-specific setting: Training, development, and test sets are form the same eventTrain: 60%, Dev = 20%, Test = 20%

2. Cross-event setting: Scenario: no labeled data for the target event. Labeled data from past events is abundant.

Cross-event: train on past events (source) and test on current event (target)

For example: Train: Nepal earthquake + Ecuador earthquakeTest: Typhoon Ruby

We use Google data assuming no past event data is available

Event-Specific Results

Cross-Event using Ecuador and Matthew as Test

Ecuador earthquake (20%) as fixed test set and all sources with 60%

Hurricane Matthew (20%) as fixed test set and all sources with 60%

Event-Specific Precision-Recall Curves and AUC

Cross-Event Precision-Recall Curves and AUC

Conclusions

• We presented results for the task of damage assessment from social media images

• We used real world datasets

• Compared non-deep learning, deep learning and transfer learning approaches

• In the event-specific case, transfer learning approach performs better

• In the cross-event case, we observed the more the data the better, same event data always helps

Thanks – Q & A@aidr_qcri