Energy Efficient and Programmable Architecture
for Wireless Vision Sensor Node
PhD Thesis presentation
MUHAMMAD IMRAN
Faculty of Science, Technology and Media,
Mid Sweden University, Sweden.
PHD THESIS ERRATA On page 6, the correct sentence is FLASH based
FPGA offers lower operating frequency.
On page 52, Figure 4-8 needs to be replaced with Fig.10
of paper IV.
On page 74, StrategyA3 output data is 421 bytes and for
StrategyA4 the output data is 411 bytes.
On page 91, the timing is 235 msec instead of 215 msec.
On page 91-92, the supply voltage is 3.6 instead of 3.5.
On page 102, paper VII and VIII paper numbering should
be interchanged.
2013-11-11 M. Imran, PhD thesis errata 2
2013-11-11 M. Imran, PhD presentation 3
OUTLINE Introduction
Challenges
Architectures analysis
Data reduction techniques
VSN with SRAM based FPGA
Conclusion
2013-11-11 M. Imran, PhD presentation 4
INTRODUCTION
2013-11-11 M. Imran, PhD presentation
Surveillance Machine vision
Camera theory research in 5th century.
Camera technology started in 20th century.
Use of camera for surveillance started in 1942.
5
TOWARDS AUTOMATION
2013-11-11 M. Imran, PhD presentation
Surveillance
Automated Surveillance Automated Machine inspection
Machine Inspection
6
CAMERA FOR INTERNET OF THINGS
Remote meter reading
2013-11-11 M. Imran, PhD presentation 7
SMART CAMERA The technological development have enabled camera
to integrate different components.
2013-11-11 M. Imran, PhD presentation
Image sensor and Lighting
Processing unit and Memory
Controlling unit and transciever
Power supply
8
COMMERCIAL SMART
CAMERAS
2013-11-11 M. Imran, PhD presentation
Manufacturer Product Processor Power FPS
Texus
Instruments NI 1722
400 MHz Freescale
PowerPC 11 W 60
Datalogic, A20 series 1,334 MIPS 12 W 60
Matrox imaging GT300 1.6 GHz processor 10 W 110
NI1722 A20 GT300
9
SMART CAMERAS BY
RESEARCH GROUPS
2013-11-11 M. Imran, PhD presentation
VSN Processing platforms Power FPS Applications
Firefly
Mosaic
ARM7TDMI (32-bit), TMEL
Atmega1281, 572.3 mW 5.2 Assisted living
MeshEye AT91SAM7S (32-bit) 290 mW 10 Surveillance
MircreEye ATMEL FPSLIC SoC (FPGA
plus microcontroller) 500 mW 15 People detection
Binary
Sensor
Flash-based FPGA, IGLOO
M1-AGL600 231 mW 5 People counting
Firefly MeshEye MireEye
10
EXISTING APPROACHES Generally researchers employ two approaches.
– Capture and process locally and transmit final results.
– Capture and transmit to server for processing.
2013-11-11 M. Imran, PhD presentation
Capturing data
Transmit data
All required vision tasks
ResultsReceive
data
VSN Server
Capturing data
Transmit results
All required vision tasks
Receive results
VSN Server
11
Partitioning processing load
among
– Software
– Hardware
– Server.
VSN ARCHITECTURES
ANALYSIS
2013-11-11 M. Imran, PhD presentation 12
TEST CASES Particle detection
Bird detection
People counting
Remote meter reading
2013-11-11 M. Imran, PhD presentation 13
ALGORITHM FLOW
2013-11-11 M. Imran, PhD presentation
MorphologySegment
Compress
Capture
Remove Bubble
Radio
Pre Processing
LabellingFeature extract
Remove Bubble
14
STRATEGY16 Data sent is approx. 114 Bytes
2013-11-11 M. Imran, PhD presentation
MorphologySegment
Compress
Capture
Remove Bubble
Radio
Pre Processing
LabellingFeature extract
Remove Bubble
SegmentCapturePre
Processing
Analysis
FPGA
SERVER
µ-controller
Transmit
VSN
MorphologyRemove Bubble
LabellingExtract
Features
15
STRATEGY36 Data sent is approx. 250KB
2013-11-11 M. Imran, PhD presentation
MorphologySegment
Compress
Capture
Remove Bubble
Radio
Pre Processing
LabellingFeature extract
Remove Bubble
Capture
MorphologyRemove Bubble
LabellingExtract
Features
FPGA
SERVER
µ-controller
Transmit
VSN
Segment
Buffer
External memory
Pre-processing
16
STRATEGY13 Data sent is approx. 500 Bytes
2013-11-11 M. Imran, PhD presentation
MorphologySegment
Compress
Capture
Remove Bubble
Radio
Pre Processing
LabellingFeature extract
Remove Bubble
SegmentCapturePre
Processing
Remove Bubble
LabellingExtract
Features
FPGA
SERVER
Morphology
µ-controller
Transmit
VSN
Binary Compression
17
LIFETIME WITH SIMULATED POWER
FLASH BASED VSN
IMPLEMENTATION
2013-11-11 M. Imran, PhD presentation
SERVERWIRELESS VISION SENSOR NODE
RadioTransceiver
Processning
SENTIO32
Camera
FLASH based FPGA
RS
232
RadioTransceiver
Flash Memory
Light
AVR32Micro-
controller
Capture
Flash ControllerSPI
Pre-processing
Buffer
Vision processing
19
LIFETIME WITH MEASURED
POWER
2013-11-11 M. Imran, PhD presentation
0
1
2
3
4
5
6
7
Life
tim
e [Y
ears
]
Time between frames [minutes]
Strategy1
Strategy2
Strategy3
Strategy4
Strategy5
Strategy6
Strategy7
Strategy8
Strategy9
Strategy10
Strategy11
Strategy12
Strategy13
Strategy16
20
Strategy13
Strategy16
SegmentCapturePre
Processing
Analysis
FPGA
SERVER
µ-controller
Transmit
VSN
MorphologyRemove Bubble
LabellingExtract
Features
SegmentCapturePre
Processing
Remove Bubble
LabellingExtract
Features
FPGA
SERVER
Morphology
µ-controller
Transmit
VSN
Binary Compression
Strategy16
Strategy13
FLASH BASED VSN
ARCHITECTURE
2013-11-11 M. Imran, PhD presentation
SERVERWIRELESS VISION SENSOR NODE
RadioTransceiver
Processning
SENTIO32
Camera
FLASH based FPGA
RS
232
RadioTransceiver
Flash Memory
Light
AVR32Micro-
controller
Capture
Flash ControllerSPI
Pre-processing
Binary compression
Morphology
Segmenta-tion
21
ENERGY CONTRIBUTION VSN
2013-11-11 M. Imran, PhD presentation
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.03
0.05
0.07
0.17
0.50
1.50
5.00
15.00
45.00
130.00
480.00
720.00
1440.00
Re
lati
ve e
ne
rgy
con
sum
pti
on
Sample period[m]
E_HARDWARE_PROC
E_COMM
E_SLEEP_PLATFORM
All required vision tasks on software
(Strategy34) All required vision tasks on hardware
(Strategy16)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.03
0.05
0.07
0.17
0.50
1.50
5.00
15.00
45.00
130.00
480.00
720.00
1440.00
Rel
ativ
e en
ergy
co
nsu
mpt
ion
Sample period [m]
E_HARDWARE_PROC
E_COMM
E_SLEEP_PLATFORM
0%10%20%30%40%50%60%70%80%90%100%
0.03
0.05
0.07
0.17
0.50
1.50
5.00
15.00
45.00
130.00
480.00
720.00
1440.00R
ela
tive
en
erg
y co
nsu
mp
tio
n
Sample Period [m]
E_SOFTWARE_PROC
E_COMM
E_SLEEP_SOFTWARE
Front end tasks and binary
compression on hardware (Strategy13)
22
Bi-level image coding
Bi-level video coding
DATA REDUCTION
2013-11-11 M. Imran, PhD presentation 23
SERVERWIRELESS VISION SENSOR NODE
RadioTransceiver
Processning
SENTIO32
Camera
FLASH based FPGA
RS
232
RadioTransceiver
Flash Memory
Light
AVR32Micro-
controller
Capture
Flash ControllerSPI
Pre-processing
Binary compression
Morphology
Segmenta-tion
BI-LEVEL IMAGE CODING
Investigated compression schemes
– G4
– G3
– JBIG2
– Rectangular
– GZIP
– GZIP_pack
– and JPEG_LS
2013-11-11 M. Imran, PhD presentation 24
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
Ene
rgy
con
sum
pti
on
[m
J]E_HARDWARE_PROC
E_COMM
NEED FOR BI-LEVEL VIDEO
CODING Data reduction beyond limit of simple image coding.
Bi-level video codec is required but not available.
Coding complexity needs to be reduced.
2013-11-11 M. Imran, PhD presentation 25
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.03
0.05
0.07
0.17
0.50
1.50
5.00
15.00
45.00
130.00
480.00
720.00
1440.00
Rel
ativ
e en
ergy
co
nsu
mpt
ion
Sample period [m]
E_HARDWARE_PROC
E_COMM
E_SLEEP_PLATFORM
Strategy13
BI-LEVEL VIDEO CODEC Bi-Level video codec functionality
– Image coding
– Change coding
– ROI coding
– Change-ROI coding
2013-11-11 M. Imran, PhD presentation
1 2 3
1 2
Change
coded image
Change
coded image
3
ROI coded
image
5
4
ROI coded
image
Change-ROI
coded image
4
Change-ROI
coded image
5
(a)
(b)
Change coding
ROI coding G4
compression
Huffman and Run
length codes Mixer
ROI row runs
ROI imageHuffman
codes
8
Bi-level data
26
CHANGE CODING Only changes are coded.
2013-11-11 M. Imran, PhD presentation
Calculate black and
white runs
Previous frame
runs access
Produce Black/
white value upto
count value
FIFO
Reset
Clk
Data
Frame_sync
Line_sync
B_runs
W_runs
Synchronization
XOR
Data_Frame
Data_Frame-1
Full/Empty
Line_sync
Frame_sync
Data
Runs
Change coding
ROI coding G4
compression
Huffman and Run
length codes Mixer
ROI row runs
ROI imageHuffman
codes
8
Bi-level data
27
ROI CODING Area of image with objects is coded.
2013-11-11 M. Imran, PhD presentation
200
300
580
730
1244
1744
2400
0
1600
24
00
Change coding
ROI coding G4
compression
Huffman and Run
length codes Mixer
ROI row runs
ROI imageHuffman
codes
8
Bi-level data
28
ROI CODING
2013-11-11 M. Imran, PhD presentation
Runs Symbols
colour
Run length
coding (8 bits)
Run length
coding (9 bits)
200 Black 200,0* 200,0*
100 White 100 100
280 Black 0,25 280
656 Black 0,0,146 0,145
200
300
580
730
1244
1744
2400
0
1600
24
00
29
G4 COMPRESSION G4 used 2-Dimensional line by line coding.
In this, the position of changing picture elements rather
than alternating black and white runs are calculated.
2013-11-11 M. Imran, PhD presentation
Reference line
Coding line
a0 a1
b1 b2
a2
Change coding
ROI coding G4
compression
Huffman and Run
length codes Mixer
ROI row runs
ROI imageHuffman
codes
8
Bi-level data
30
1716 Bytes
2013-11-11 M. Imran, PhD presentation
250 Kbytes
500 Bytes
223 Bytes
125 Bytes
33 Bytes
Input Images
Image coding
Change coding
ROI coding
Change-ROI coding
Particle detection Meter reading Bird detection People counting
250 Kbytes
585 Bytes
203 Bytes
150 Bytes
65 Bytes
768 Kbytes
411 Bytes
249 Bytes
667 Bytes
517 Bytes
200 Kbytes
2060 Bytes
2552 Bytes
2605 Bytes
31
VSN WITH BI-LEVEL VIDEO
CODEC
2013-11-11 M. Imran, PhD presentation
SegmentPre-
processingCapture
Background image
Morphology
SENTIO32
Camera
FPGASPI
Buffer
RS
232
AVR32
Bi-level Video coding
LED ring
Radio
Battery
Memory controller
FLASH memory
32
COMPARISON OF SYSTEMS
Reduced energy consumption of 1.5 to 376 times.
Reduced output data by a factor of approximately 3 to 246.
2013-11-11 M. Imran, PhD presentation
Applications Systems E_Proc
(mJ)
E_Comm
(mJ)
Avg.
Output data
(bytes)
Bits/
pixel
Max.
Freq.
(FPS)
Particle
detection Published [8] 0.63 3.52 500 0.016 60
Proposed 2.04 0.38 33 0.001 100
Meter
reading Published [2] 430.3 207.9 6912 0.500 0.04
Published_scaled [2] 515.3 478.3 16000 0.500 0.03
Proposed 2.04 0.60 65 0.002 100
Bird
detection Conventional 6.03 4.9 703 0.007 33
Proposed 5.76 1.83 249 0.003 33
People
counting Published [3] 0.004 4.29 2048 2.000 15
Published_scaled [3] 0.08 27.06 51200 2.000 5
Proposed 1.57 11.6 1715 0.067 18
33
SRAM based FPGA for duty
cycling
Low complexity background
subtraction
FPGA BASED VSN: SENTIOF-CAM
2013-11-11 M. Imran, PhD presentation 34
LOW COMPLEXITY
BACKGROUND
2013-11-11 M. Imran, PhD presentation
Image sensorScaled down
Internal Memroy
Scaled up
Microcontroller/FLASH Memory
CAM/FLASH
-
First time or when needed
FPGA
Splitter
CAM/FLASHCurrent Pixels
Background Pixels
35
VSN ARCHITECTURE
2013-11-11 M. Imran, PhD presentation
Camera
FPGA
SP
I
AVR32
LED ring
Radio
Battery
Scaled down
Scaled up
Memory buffer
Flash memory
ROI
CC Mixer
Compress
Bi-level video codecBackground storage model
Flash controller
SegmentPre-
processingCapture Morphology
Front end tasks
36
SETUP: SENTIOF-CAM
2013-11-11 M. Imran, PhD presentation
A B C D
0 10 20 30 40 50 60 700
200
400
600
800
1000
1200
X: 64.5
Y: 0.09596
Time (ms)
Insta
nta
neous C
urr
ent
(mA
)
One cycle instantaneous current:meter reading
transmission
wakeup statesleep
state
sleep-to-wakeup state
FPGA
power-on
FPGA configurationprocessing
sleep
state
SENTIO32 SENTIOF-CAM GUI and received data FOV GUI
Measurement setup Instantaneous current for a duty cycle
37
CURRENT MEASUREMENT
FOR DUTY CYCLE
2013-11-11 M. Imran, PhD presentation
0 10 20 30 40 50 60 700
200
400
600
800
1000
1200
X: 64.5
Y: 0.09596
Time (ms)
Insta
nta
neous C
urr
ent
(mA
)
One cycle instantaneous current:meter reading
transmission
wakeup statesleep
state
sleep-to-wakeup state
FPGA
power-on
FPGA configurationprocessing
sleep
state
38
POWER AND TIME OF
DIFFERENT PROCESSES
2013-11-11 M. Imran, PhD presentation
Applications Config.
time
(ms)
Config.
power
(W)
Proc.
time
(ms)
Proc.
power
(W)
Comm.
Time
(ms)
Comm.
power
(W)
Total
energy
(mJ)
Particle detection 23.6
0.12
21
0.67 3.2 0.13 17.2
Meter reading 23.6
0.12
21
0.67 4.3 0.13 17.4
39
LIFETIME
2013-11-11 M. Imran, PhD presentation
0
2
4
6
8
10
12
14
Life
tim
e [y
ear
s]
Sample period [m]
PUBLISHED_PARTICLE
MEASURED_PARTICLE
PUBLISHED_METER
MEASURED_METER
40
An energy efficient and programmable VSN is developed and
implemented.
– Low complexity
– Generic architecture
To reduce communication energy on hardware implemented
VSN, a bi-level video coding is developed and implemented.
A VSN with proposed Bi-level video coding reduced energy
consumption by a factor of 1.5 to 376.
A VSN with SRAM based FPGA has been evaluated for duty
cycle applications.
– Depending on application a lifetime of 3.2 years with 37 kJ energy (AA).
CONCLUSION
2013-11-11 M. Imran, PhD presentation 41
FUTURE WORK This study can be extended for applications which require
greyscale or colour data for classification.
Bi-level video coding which handle movement of objects.
A taxonomy which has been proposed can be improved by
making it more exhaustive.
Wireless smart camera technology can be integrated for Internet
of things.
2013-11-11 M. Imran, PhD presentation 42
–Prof. Mattias O’Nils
–Najeem Lawal
–Prof. Bernhard Rinner
–Committee members
–All audience
THANKS!
2013-11-11 M. Imran, PhD presentation 43
http://www.orissadiary.com
http://lifeboat.com/ex/security .preserver
http://www.epicsysinc.com/bl og/machine_vision_history
http://edenprairieweblogs.org/scottneal/page/45/
http://www.marketingsavant.com/2011/11/getting-started-in-a-
social-media-career-part-i/
REFERENCES
2013-11-11 44 M. Imran, PhD presentation