ICE Summer School, July 2013

Visual Sensor Networks

Bernhard Rinner

Institut für Vernetzte und Eingebettete Systeme

B. Rinner. 2

Agenda

Sensor Networks

Smart Cameras

Visual Sensor Networks

B. Rinner 3 3 3

Introduction to Sensor Networks

B. Rinner 4

Wireless Sensor Networks (WSNs)

• Networks of typically small, battery-powered, wireless devices, (“sensor nodes”, “motes”) – On-board processing,

– Communication, and

– Sensing capabilities.

Sensors

Processor

Radio

Storage

P O W E R

Sensor node schematics [© Oracle Labs]

B. Rinner 5

Sensor Node Platforms

• From research prototypes to commercial products

The Vision „Smart Dust“ UC Berkeley late 1990‘s

Commercial Products „Mote-on-a Chip“ Dust Networks, 2010

Research Prototypes „Mica-2“ Crossbow 2004

B. Rinner 6

Some Applications of Sensor Networks

• Health

• Structural Monitoring

• Agriculture

• Environmental

[(c) University of Ghent] [Kim et al. ACM SenSys, 2006

[AgriNet] [M. Welsh, Harvard 2007]

B. Rinner 7

Communication is Key

• Wireless communication is an enabling technology – Eases deployment

– Enables mobility

– Increases flexibility

– Reduces costs

• Communicate on demand (ad hoc, spontaneous) with dynamic infrastructure – Nodes organize themselves into network

– Data is transferred via multiple hops

source destination

B. Rinner 8

But also …

• Advances in sensor technology – Micro electro-mechanical system

(MEMS) revolutionized sensing

– Integration of mechanical and electrical components on single chip

– Example: 3D accelerometer (in your cell phone)

• Embedded processors and integration – Moore’s Law still valid

– Trade-off between processing performance and power consumption

[© SensorDynamics]

[© ARM]

B. Rinner 9 9 9

The “Energy Problem”

• Major sources of energy consumption – Sensing, computing, communication

– High temporal variation

• Energy is the scarce resource for WSN. Several challenges – What energy reservoirs to exploit?

Constraints: availability, max. power, size, …

– How to distribute power over the network? Energy provider and consumer might be dislocated.

– How to control the distribution? “The proper amount of energy in the right place at the right time”

• Sensor networks have always been a “green” technology!

B. Rinner 10

Alternative Energy Reservoirs

• Maybe micro-heat engines – Exploit MEMS technology

to build „internal combustion engines“

– Expected power: 10-20 W

– Still in early research/development phase

• Or harvest energy from environment – Organic semiconductors for exploiting indoor ambient light

– Thin film batteries for storing energy

– EnHANTs : energy harvesting networked active tags

[Handbook of Sensor Networks, Wiley]

[Columbia University, CLUE]

B. Rinner. 11 11 11

Smart Cameras

B. Rinner. 12

Principle of Smart Cameras

• Smart cameras combine – sensing,

– processing and

– communication

in a single embedded device

• perform image and video analysis in real-time closely located at the sensor and transfer only the results

• collaborate with other cameras in the network

TrustEYE.M4 prototype on top of RaspberryPI

B. Rinner. 13

Differences to traditional Cameras

Traditional Camera – Optics and sensor

– Electronics

– Interfaces

delivers data in form of (encoded) images and videos, respectively

Smart Camera – Optics and sensor

– Onboard computer

– Interfaces

delivers abstracted image data and is configurable and programmable

Sensor

Electronics

Image enhancement/ Compression

Image Video

Sensor

Embedded Computer

Image analysis

„Events“

Programming Configuration

Light Light

B. Rinner. 14

SmartCams look for important things

• Examples for abstracted image data – compressed images and videos

– features

– detected events

© CMU

B. Rinner 15

Be aware of scarce Resources

• Major resource limitations – Processing power

– Communication bandwidth

– Onboard memory

– Energy

• Various Prototypes (with decreasing performance)

Sony XCISX100C/XP x86 VIA Eden ULV @ 1 GHz

TrustEYE.M4 ARM Cortex@ 168MHz

SLR Engineering Atom Z530@ 1.6 GHz

CITRIC PXA 270@ 13-640MHz

[Rinner et al. The Evolution from Single to Pervasive Smart Cameras. Proc. ICDSC 2008]

B. Rinner 16 16 16

Characteristics of Visual Sensor Networks

B. Rinner 17

Video Surveillance Network

• 3rd generation – all-digital systems

• 3+ generation – smart cameras

– surveillance tasks run on-site on smart cameras, e.g., • video compression traffic statistics

• accident detection wrong-way drivers

• stationary vehicles (tunnels) vehicle tracking

• 1st and 2nd generation – primarily analog frontends

– backend systems are digital

[Regazzoni, Ramesh, Foresti. Special Issue on Video Communications, Processing and Understanding for Third Generation Surveillance Systems. Proceedings of the IEEE. October 2001]

B. Rinner 18

Video Surveillance Network (2)

• Even third generation networks rely on “heavy” infrastructure. – Camera nodes: sensor, onboard processing (encryption)

– Network: hierarchically structured, wired, large bandwidth

– Energy: dedicated supply

• Surveillance networks typically consist of large number of cameras

• Processing in network is fixed; (compressed) data is streamed to control center

B. Rinner 19

Characteristics of VSN

• Visual sensor networks lie somewhere in between wireless sensor networks (WSN) and multi-camera/surveillance networks.

• VSN have unique characteristics (wrt. traditional WSN)

• Resource limitations – Need to process and transfer large amounts of data

– Energy and bandwidth

• On-board processing (cp. Smart cameras) – Challenging vision algorithms

– Adaptive behavior

[Soro et al. A Survey of Visual Sensor Networks. Advances in Multimedia 2009]

B. Rinner 20

Characteristics of VSN (2)

• Real-time operation – Most applications require real-time analysis (camera to user)

• Location and orientation information (spatial calibration) – Absolute or relative coordinates and orientations

– (Multi-)camera calibration

• Time Synchronization (temporal calibration)

• Data Storage – Access to historic data necessary, eg., frame buffer, detected events

– Stored data may be discarded over time

• Autonomous Camera Collaboration – cp. Distributed smart cameras (DSCs)

B. Rinner. 21

(Selected) VSN Problems

• Sensor Placement – Eg., dynamic setting of PTZ parameters

• Clustering, cluster head election – Eg., what cameras should “work together”, who is the “leader”

• Synchronization and calibration – Eg., establish temporal and spatial correlation

• Data (and energy) distribution – Eg., when and what data to exchange

• …

B. Rinner. 22 22 22

Coordinating Resources in

Visual Sensor Networks

B. Rinner 23

Configuring Smart Camera Networks

• Smart camera networks process data onboard can modify their functionality/execute actions during runtime to reflect changes – to the state of the environment

– to the user criteria

• A configuration describes what is processed/executed where; specified by – Description of camera network (including the available actions/tasks)

– Specification of the objective

• We study configuration methods to use scarce resources in these networks more efficiently

B. Rinner 24

Configuration Problem (example)

• Configuring a camera network – Select a set of cameras to monitor an area of interest

– Set the sensor (frame rate, resolution, PTZ) to achieve QoS

– Assign monitoring functions to cameras

– Optimize wrt. multiple criteria

– Dependent on dynamics of environment

s1

s2 s3

s4

s5

t1

t2

t3

p1, p2

p3

p4, p5

B. Rinner 25

Configuration Design Space

Design space for configuration methods is given by:

• Dynamics of environment (static vs. mobile observation points)

• Configuration algorithm (centralized vs. distributed)

• Tasks and sensors (homogeneous/heterogeneous; static/mobile cameras)

• A priori knowledge (complete vs. no knowledge of environment/VSN)

• Various alternatives for solving this optimization problem, eg. – Centralized configuration algorithms

– Distributed configuration algorithms

focus on resource-aware approaches

B. Rinner 26

Centralized Configuration with EA

• Approximation with evolutionary algorithm satisfying all requirements along multiple criteria (eg., energy, data, QoS)

• Smart Camera Network – Set of cameras at known position with fixed FoV

– Sensor configurations (frame rate, resolution)

• Observation Area – Static set of observation points with monitoring activity a

at required QoS (pot, fps)

• Monitoring tasks – Assign procedures for achieving

– Required resources for

},...{ 1 nSSS =

[Dieber, Micheloni, Rinner. Resource-Aware Coverage and Task Assignment in Visual Sensor Networks IEEE Transactions on Circuits and Systems for Video Technology, Aug 2011]

},...,{ 1 ki ddD =

},...,{ 1 mttT =

},...,{; 1 apaa ppPAa =∈),,(),( iiiii emcdPr →

B. Rinner 27

Self-aware Configuration

• Adopted from proprioceptive computing systems – use proprioceptive sensors to monitor “one self”

(concept from psychology, robotics/prosthetics, …, fiction)

– reason about their behavior (self-awareness)

– effectively and autonomously adapt their behavior to changing conditions (self-expression)

• Demonstrate autonomous multi-object tracking in camera network – Exploit single camera object detector & tracker

– Perform camera handover

– Learn camera topology

B. Rinner 28

Bid C4

Virtual Market-based Handover

• Initialize auctions for exchanging tracking responsibilities – Cameras act as self-interested agents, i.e., maximize their own utility

– Selling camera (where object is leaving FOV) opens the auction

– Other cameras return bids with price corresponding to “tracking” confidence

– Camera with highest bid continues tracking; trading based on Vickrey auction

Camera 1 Camera 2

Camera 3

Camera 4

Init auction

Bid C3

Fully distributed approach no a-priori topology knowledge required

B. Rinner 29

Market-based Tracking Handover

• Utility function (each camera)

rpjvcOUiOj

ijjii +−Φ⋅⋅= ∑∈

)]([)(

tracking decision visibility confidence payments made payments received

Simulation green: tracking yellow: shared FOV red: trading (handover)

B. Rinner 30

Tracking Performance

• Tradeoff between utility and communication effort

Scenario 1 (5 cameras, few objects) Scenario 2 (15 cameras, many objects)

• Emerging Pareto front [Esterle et al. Socio-Economic Vision Graph Generation and Handover in Distributed Smart Camera Networks. ACM Trans. Sensor Networks. 2013]

B. Rinner 31

Learn Neighborhood Relationships

• Gaining knowledge about the network topology (vision graph) by exploiting the trading activities

• Temporal evolution of the vision graph

B. Rinner 32

Learning Heterogeneous Strategies

• Heterogeneous strategies at cameras may improve Pareto front

• Adapt camera behaviour by online learning using bandit solvers

Homogeneous vs. heterogeneous handover strategies (offline)

Online learning strategies with different bandit solvers

[Lewis et al. Learning to be different: Heterogeneity and Efficiency in Distributed Smart Camera Networks. In Proc. IEEE SASO. 2013]

B. Rinner 33 33 33

Applications

B. Rinner 34 34 34

#1 Trustworthy Cameras

• Smart cameras – Highly capable embedded systems (on-board video analysis)

– Large software stacks

– Networked devices using closed (CCTV) and public networks

• Applications no longer only in public but also in private areas (assisted living, home monitoring, …)

• Protection of sensitive image data – Protection against manipulation (e.g., enforcement applications;

evidence at court)

– Privacy of monitored people

B. Rinner 35 35 35

Goals and Assumptions

• We present a system level approach that addresses the following security issues: – Integrity: detect manipulation of image and video data

– Authenticity: provide evidence about the origin of image and videos

– Confidentiality: make sure that privacy sensitive image data cannot be accessed by an unauthorized party

– Multi-level Access Control: support different abstraction levels and enforce access control for confidential data

• Security and privacy protection as inherent features of the camera

• Considered attack types: only software attacks [Winkler, Rinner. Securing Embedded Smart Cameras with Trusted Computing.

EURASIP Journal on Wireless Communications and Networking, 2011 ]

Approach

• Bringing of Trusted Computing concepts into cameras • Trusted Platform Modules (TPMs) are well defined, readily

available and cheap

• TC is an open industry standard • TPMs are available from many manufacturers

B. Rinner 36

Hardware Security Anchor

37

• Trusted Platform Module (TPM) at a glance – Secure storage for cryptographic keys – Data encryption, digital signatures – System status monitoring and reporting (measurement + attestation) – Unique platform ID

Security Chip (TPM) Image Sensor CPU RAM

Bootloader

Operating System (e.g., Embedded Linux)

Software Libraries and Middleware

Image Processing and Analysis Communication …

Software

Hardware

B. Rinner

Implemented Security Features

38

• Trusted boot where camera software stack is “measured” and the status is securely reported to operator

• Integrity and authenticity guarantees using non-migratable, TPM-protected RSA keys

• Freshness/timestamping for outgoing images via TPM-protected tick (counter) sessions

B. Rinner

B. Rinner 39 39 39

Hardware Prototype

• TI OMAP 3530 CPU: ARM @ 480MHz and

DSP @ 430MHz

• 256MB RAM, SD-Card as mass storage

• VGA color image sensor

• wireless: 802.11b/g WiFi

and 802.15.4 (XBee)

• LAN via USB (primarily used for debugging)

• Atmel hardware TPM

on I2C bus

Privacy Protection Approaches

40

• Protection as an inherent feature of the camera

• Object-based protection: Identification of sensitive data (e.g., human faces)

• Data abstraction and obfuscation

• Global protection techniques: Uniform protection of entire

frames (insensitive to misdetections of computer vision)

B. Rinner

Multi-Level Protection

41

• Video stream contains sub streams • Every sub stream is encrypted

– Hardware-bound cryptographic keys

• Recovery of identities only via four eyes principle

Video Stream Smart Camera Sub Streams

B. Rinner

High-Level Processing Flow

42 B. Rinner

B. Rinner 43 43 43

Privacy-aware Camera Networks

• What about users (i.e., monitored people)? • users usually do not care much about integrity, authenticity of time

stamping

• users (hopefully!) care about confidentiality and privacy!

• Question 1: How can we increase privacy awareness?

• Question 2: How can we demonstrate that (our) cameras protect the privacy of users?

B. Rinner 44 44 44

Raising Privacy Awareness

• Let users know if there are cameras in their environment

• Use user's handheld (e.g., smart phone) for location-based notifications

User Feedback

• Goal: Trustworthy feedback to monitored persons about camera’s privacy protection

• Visual communication for authentication – Direct line of sight – Intuitive way to select intended camera

• Operator discloses applications to TrustCenter

T. Winkler and B. Rinner, “User Centric Privacy Awareness in Video Surveillance,” Multimedia Systems Journal, vol. 18, no. 2, pp. 99–121, 2012.

B. Rinner

Attestation Report

46 B. Rinner B. Rinner

B. Rinner 47

#2 Aerial Cameras for Disaster Mgm.

• Develop autonomous multi-UAV system for aerial reconnaissance

• Up-to-date aerial overview images are helpful in many situations: “Google Earth with up-to-date images in high resolution”

• Small-scale quadcopter platform with onboard sensors and computation

• GPS receiver for autonomous waypoint flights

• Generic framework not bound to specific UAV

B. Rinner 48

Key Challenges

• Increase autonomy – Control and coordination of multiple UAVs

– High-level interaction with user

• Provide prompt response to user – Provide preliminary results fast and improve over time

• Deal with strong resource limitations – Flight time, payload, computation and communication

– Limited sensing capabilities

B. Rinner 49

Autonomous UAV Operation

Mission Planning Flight

Real-World

Simulator

Image Analysis

scenario specification

waypoints

Single/multiple UAV

captured image/video

stiching, detection

user interface

B. Rinner 50

Key Questions

• How to generate and update movement routes for the UAVs? – Achieve multiple optimization goals

– Deal with changes in the environment

• How to setup a wireless UAV network? – Provide networking coverage

• How to generate the mosaic image? – Apply incremental image stiching

– Combine RGB and thermal images

• System integration and demonstration

51

Demonstration Video

B. Rinner

B. Rinner. SCVSN Tutorial (Chapter 3) 52 52 52

Research Directions of Visual Sensor Networks

B. Rinner. 53

#1: Architecture

• Low-power (high performance) camera nodes – Dedicated platforms: vision processors, PCBs, systems

– Many examples: CITRIC, NXP

• Visual/Multimedia Sensor Networks – Topology and (multi-tier) architecture

– Multi-radio communication

• Dynamic Power Management – For sensing, processing and communication

How to design resource-aware nodes and networks

B. Rinner. 54

#2: Networking

• Ad hoc, p2p communication over wireless channels – Providing RT and QoS

– Eventing and/or streaming

• Dynamic resource management – (local) computation, compression, communication, etc.

– Degree of autonomy: dynamic, adaptive, self-organizing

– Fault tolerance, scalability

– Network-level software, middleware

How to process and transfer data in the network

B. Rinner 55

#3: Deployment, Operation, Maintenance

• Development support for applications – Model/simulate the application (function, resources, QoS)

– Reuse/exchange of software/libraries

– Software updates, debugging etc.

• Autonomous calibration and scene adaption – Avoid manual procedures

– Adapt to different scenes and settings

• Network configuration

Consider the entire life cycle of the camera network

B. Rinner. 56

#4: Distributed Sensing & Processing

• Sensor placement, calibration & selection – Optimization problem

– Distributed approaches eg., consensus, game theory, multi-agent systems

• Compressive Sensing

• Collaborative data analysis – Multi-view, multi-temporal, multi-modal

– Sensor fusion

• Online/real-time processing – Can not effort to store large amounts of data

Where to place sensors and analyze the data

B. Rinner 57

#5: Mobility

• Mobile cameras are ubiquitous – PTZ, vehicles, robotics etc.

– Mobile phones

• Advanced vision algorithms – Ego motion, online calibration

– Closed-loop control, active vision

How to exploit networks of mobile cameras

B. Rinner 58

#6: Usability

• Ease of deployment, maintenance – Self-* functionality

– “Smart cameras for dumb people”

• Privacy and Security – Trust of the user

– Control the privacy setting

• Interaction with the camera network

How to provide useful services to people

B. Rinner 59

#7: Applications

• Demonstrations – Large scale networks eg., for surveillance

– Small scale networks eg., for entertainment, home environments

– Only single camera application?

• Market opportunities

• Killer Application

What applications can (only) be solved by DSC

B. Rinner. SCVSN Tutorial (Chapter 3) 60

Summary

• VSNs exploit various advantages of distributed camera sensors such as increased coverage, redundancy and 3D information.

• Distributed cameras impose various challenges such as huge amount of data, required infrastructure and (network) topology.

• VSN have unique characteristics (wireless sensor networks vs. surveillance camera networks)

• Current research addresses signal processing, communications, architecture and middleware issues.

B. Rinner 61

Acknowledgements & Further Info

Pervasive Computing @ AAU UAV Research http://pervasive.aau.at http://uav.aau.at

• Tutorial site Most recent course material is available at

http://pervasive.aau.at/S5-tutorial