© Fraunhofer

Smart Video Surveillance and Privacy

Dr.-Ing. Erik Krempel,

Fraunhofer IOSB, Karlsruhe, Germany

© Fraunhofer IOSB 2

What can we do with video analytics…

SMART VIDEO SURVEILLANCE

© Fraunhofer IOSB 3

From Visual to Symbolic Video Surveillance

• The classic approach of video surveillance is visual videomonitoring by security staff.

• Today, video analytics are more and more in use as assistance systems. Mainly to attract and sustain attention of the staff to possible points of interst.

• Current research is going towards useage of advanced video analytics for symbolic situation awareness and „privacy-aware management by exception“

Classic Approach Intelligent Video Analytics

Privacy-aware Situation

Awareness Tools

© Fraunhofer IOSB 4

Video Analytics for Scene Understanding

Object Tracking

Activity RecognitionSemantic SceneUnderstanding

Scene Context Recognition

Object DetectionImage Enhancement

© Fraunhofer IOSB 5

Current Research: Advanced Objekt Detection, Tracking and Re-Identification

• Multi-camera multi-object detection and tracking of persons and vehicles

• indoor / outdoor applications with complex illuminationconditions

• self-calibrating cameras (automated geometric and colorcalibration)

• New approaches on dynamic learning of objects‘ unique features

© Fraunhofer IOSB 6

Towards Activity Recognition for (Global) Scene Understanding

• Classification of activities in videos.

© Fraunhofer IOSB 7

Person Retrieval

• Component that is looking for similar persons

in archive in a specific time interval

• Features of the person of interest (color features and texture features) are compared to already computed features of other tracks

• System generates a listof tracks, sorted basedon their similarity tothe person of interest

• Operator can verify the results visually

© Fraunhofer IOSB 8

Why do we need to certify smart video surveillance systems

CERTIFICATION

© Fraunhofer IOSB 9

Security

What is the error rate of video processing algorithms?

False positive rates

False negative rates

Is the system robust against environment conditions?

Rain on the camera

Light conditions

Is the system safe against attacks?

The systems keeps secrets for itself (Confidentiality)

The systems stays operational in the presence of an attacker (Availability)

The systems detects the same results in the presence of an attacker (Integrity)

© Fraunhofer IOSB 10

Trust / Transparency

© Fraunhofer IOSB 11

Trust

How can you achieve real transparency in such complex systems?

Is there a human preventing the system from making wrong decisions?

Does the systems prevent discrimination?

© Axis Communications

© Fraunhofer IOSB 12

Efficiency dimension

Is the system able to reduce crime / increase security?

Is the system easy to use?

Can you extend the system / use components of another manufacturer?

© Fraunhofer IOSB 13

Freedom Infringement

Does the system prevent discrimination?

Are video processing algorithms restricted to legal applications?

Does the system prevent misuse by the operator?

Security of the data processing…

Secure against data theft…

Purpose limitation…

Errors by the systems…

© Fraunhofer IOSB 14

RESEARCH WORKPrivacy in smart video surveillance

© Fraunhofer IOSB 15

Situation-dependent video surveillance

Alexander Roßnagel, Monika Desoi, and Gerrit Hornung (2011)

© Fraunhofer IOSB 16

Idea: Privacy-aware Surveillance Workflows

Default Mode: Optimized for privacy Assessment Mode:Event-specific, privacy preserving

Eventdetected

Eventconfirmed

Eventresolved

Investigation Mode:Event specific functions unlocked

© Fraunhofer IOSB 17

Default Mode: Optimized for Privacy

Computer vision algorithms in background

Abstract representation of observed environment visualized

No access to video data

No access to archived data

Exposes as little information about observed persons as possible

© Fraunhofer IOSB 18

Assessment Mode: Event-specific, Privacy Preserving

Eventdetected

Assessment view according to event type

Anonymized live video data released

Possibly anonymized access to a limited buffer of recorded (video) data

Only selected cameras

© Fraunhofer IOSB 19

Investigation Mode: Event-specific Functions Unlocked

Eventconfirmed

Eventresolved

Allows additional privacy intrusions for investigation purposes

I.e., retrieving the person who dropped a piece of luggage

Restrict additional analyses to persons related to the event under investigation

© Fraunhofer IOSB 20

IMPLEMENTATIONHow to build systems with this concept…

© Fraunhofer IOSB 21

Security Use-Case

© Fraunhofer IOSB 22

Safety Use-Case: Prototype NurseEye

© Fraunhofer IOSB 23

Operator interaction

Default mode:

No Access

Assessment mode:

Access to anonymized video

Investigation mode:

Full access to video to help in emergency handling

© Fraunhofer IOSB 24

Transparency

All cameras come with displays showing the current mode and data usage

Displays become monitor for chat in the investigation mode

Default mode Assessment mode Investigation mode

© Fraunhofer IOSB 25

Video NurseEye

© Fraunhofer IOSB 26

TOOLS

© Fraunhofer IOSB 27

User Study on Anonymization Techniques: Introduction

Which obfuscation technique should be used?

Regarding privacy (identity leakage), utility and perceptual video quality

ROI Anonymized person

Blurring Silhouette Edge detection PixelizationBlurring(gray scale)

© Fraunhofer IOSB 28

Results: Subjective assessments

1 = silhouette; 2 = pixelization; 3,5 = edge detection; 4 gray scale blurring; 6-8 color blurring

Error bars represent one standarddeviation of the data sample

Perceived privacy protection and perceived image quality

© Fraunhofer IOSB 29

Video anonymisation

Edge detectionPixelization

Gaussian blurring Silhouette

© Fraunhofer IOSB 30

Adaptive „Privacy-Masking“ for Pan/Tilt/Zoom Camera

• Advanced Privacy Masking for semi-stationary PTZ-cameras

• Given position of camera over ground (or an elevation model of the site) optical distance toobjects in the scene is estimated

• Given a parameter for „privacy-preserving resolution“ video is pixelized or blurredinhomoge-nously depending on distance to object.

• Highly senstive areas (e.g. buildings / windows) can be blacked out of the stream bydynamic adaptable polygons (depending on pan/tilt/zoom settings)

© Fraunhofer IOSB 31

Information Flow Tracking in NEST

Video &Observation

archive

PEP

HMIGeoViewer

Observations

PEP

Requests

HMIStreamViewer

PEP

Imageexploitationalgorithm

Imageexploitationalgorithm

Imageexploitationalgorithm

PEPIMG

PolicyDecision

Point

PolicyInformation

Point

Policy

Observations

Control

PEP

Acce

ss Co

ntr

ol

OOWM

Association

PEP HLS-Alarms

PEP Classification

PEP Fusion

© Fraunhofer IOSB 32

Discussion

“Data is not an asset, it’s a liability”-Marko Karppinen-

Contact Information:Erik Krempel

Fraunhofer IOSBFraunhoferstr. 176131 Karlsruhe, Germany

erik.krempel@iosb.fraunhofer.de+49-721-6091-292