1 Dept. of Electrical and Computer Engineering University of Missouri, Columbia, MO Page 1 Power-Rate-Distortion Analysis for Energy Minimization of Video Coding Prof. Zhihai (Henry) He Electrical and Computer Engineering University of Missouri Columbia MO 65203 [email protected](573) 882-3495 Dept. of Electrical and Computer Engineering University of Missouri, Columbia, MO Page 2 Outline 1. The need for energy minimization of video encoding • Related research projects in wireless video sensor networks 2. Power-rate-distortion analysis 3. Energy minimization using power-rate-distortion analysis 4. Results 5. Further Research Issues and potential collaborations
Transcript
1
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 1
Power-Rate-Distortion Analysis for Energy Minimization of Video Coding
Prof. Zhihai (Henry) HeElectrical and Computer EngineeringUniversity of MissouriColumbia MO 65203
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 2
Outline
1. The need for energy minimization of video encoding• Related research projects in wireless video sensor
networks2. Power-rate-distortion analysis3. Energy minimization using power-rate-distortion analysis4. Results5. Further Research Issues and potential collaborations
2
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 3
Trend in Information Technology
Smaller& faster
MobileComputing
Wireless
MassiveDynamic
On-demand
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 4
Wireless Video Sensor Network
1. Vision is the dominant approach for people to receive information
2. Images and videos are a critical source of information for situational awareness and decision making.
3. Image and video sensors are playing an increasingly important role in
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 5
Wireless Sensor NetworkWireless Sensor Network
A wireless video sensor network is a group of spatially distributed video sensor nodes that communicate with each other over a dynamic wireless network to collect visual information about the physical environment.
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
detection. (People fell and laid there for hours unattended)
5. Peace in mind for family members (summarized video message)
For Automated Functional Assessment and Safety Enhancement
Wireless Sensor NetworksNational Science Foundation
NIHFederal AOA
5
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 9
For Automated Functional Assessment and Safety Enhancement
Wireless Sensor Networks
Sensors1. Motion2. Bed /sleepless3. Gait4. Temperature5. Door6. Video
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 10
For Automated Functional Assessment and Safety Enhancement
Wireless Sensor Networks
Social Issues
Video communicationSystem Design
Long-lived Video Monitoring System
1. Low-power wireless video monitoring system (lifetime >= a month unobstrusive Make them feel comfortable)
2. Flexible, non-invasive, and rapid deployable,
Two Major Research Issues
6
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 11
• Converting massive visual information into a compact database.
• Target: Reviewing hundreds hours of video within an hour.
For Automated Functional Assessment and Safety Enhancement
Wireless Sensor Networks
Visualization and Mining of Massive Video Data
Wireless Video Camera
Full-time caregiver
Automated Functional Assessment
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 12
for wildlife monitoring, behavior analysis and disease tracking
1. Design a wireless video sensor network to collect important visual, motion, and location information about animals’ activities for behavior analysis.
2. Part of the US National Ecology Observation Network (NEON)
Wireless Video Sensor NetworkingNational Science Foundation
7
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 13
Princeton ZebraNetOnly knowing position is not enoughDo not know:
• What is animal doing?• Why is it doing like this?
(environmental context)• Interaction or not.
Wildlife disease propagates through interactions
Need visual information See what the animals see in the field!
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 14
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
1. A new generation of wildlife tracking technologies
2. Allow us to collect new types of scientific data and enable a host of new wildlife research opportunities.
8
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 15
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
Sensor Device(Mote)
Embedded DSPCompression)
Battery Set
Camera
Antenna for Retrieval
Drop-OffControl Unit
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 16
PondFenceStation
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
Missouri Dept.Of Conservation
9
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 17
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
Acceleration Sensor Data
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 18
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
• Acceleration data• GPS• Physiology sensors
Video Data
Scene classification
Fusion
• Activity statistics (walking, running, feeding, bedding, etc) at different time and environments
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 19
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 20
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
Citizen-Science for Nation-Wide Wildlife Video Monitoring
Working with New York State Museum and NSF NEON program, Missouri Department of Conservation
WildTube On-going…….
11
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 21
for wildlife monitoring, behavior analysis and disease tracking
Wireless Video Sensor Networking
Problems in Video Compression
1. Energy minimization. • Target: monitoring the animal behavior for a season, 3 months.• Video encoding: computationally intensive and energy-demanding
Visualization and Mining of Massive Video Data
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 22
for aerial video surveillance
Wireless Video Sensor Networking
Images and videos have become an important source of information for intelligence gathering, situational awareness, and decision making.
Unmanned aerial and ground vehicles (UAV and UGV), image and video sensors are extensively used in surveillance and security applications.
National Geospatial IntelligenceAgency
12
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 23
Outline
1. The need for energy minimization of video encoding• Related research projects in wireless video sensor
networks2. Power-rate-distortion analysis3. Energy minimization using power-rate-distortion analysis4. Results5. Further Research Issues
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 24
Conventional Video Encoding Portable Video Encoding
• Rate-distortion analysis
• Minimize distortion under rate constraint
• Power Rate-distortion analysis
• Minimize distortion under rate and power constraints.
Power-Rate-Distortion Analysis
1. Storage space / bandwidth: increased hundreds of times2. Battery lifetime: progress much slower.
“the biggest impediment to our technological future isn’t Moore’s law, ……it is battery life!” Wired Magazine, 2004.
13
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 25
Power-Rate-Distortion Analysis
• Extend the traditional rate-distortion (R-D) analysis to power-rate-distortion (P-R-D) analysis.
• 1-D more flexibility in energy minimization.• Trade-off between
• Using this technology, we can significantly extend the operational lifetime of portable video communication devices.
R-D
P-R-D
Bits
Computation
Energy
Bits Energy
Computation Communication
Major Results
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 26
From R-D to P-R-D
R-D P-R-D
Incorporate the third dimension, to extend the classicalR-D analysis to Power-R-D analysis for energy-aware video encoding.
For a given configuration of bit and energy resources, what is the minimum source coding distortion we can achieve?
D(R) D(R, P)
14
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 27
1. How to design an energy-scalable video encoder? (Control)
2. How to model its P-R-D behavior? (Modeling)
3. How to save energy in video compression? (Optimization)
4. What is the performance limit of a wireless video sensor under the bit and energy resource constraints? (Analysis)
Major Research Issues
Power-Rate-Distortion Analysis
Information-Theoretic Study Too challengingResort to an operational approach: design analysis optimization
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 28
Step 1: Introduce a set of complexity control parameters, , to control the complexity of major encoding
modules.
P-R-D Video Encoder Design
15
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 29
Step 1: Introduce a set of complexity control parameters, , to control the complexity of major encoding
modules.Step 2: The encoding complexity C is then a function of ,
, denoted by C
P-R-D Video Encoder Design
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 30
Step 1: Introduce a set of complexity control parameters, , to control the complexity of major encoding
modules.Step 2: The encoding complexity C is then a function of ,
, denoted by C
With Dynamic Voltage Scaling (DVS), a power control technology for circuit system design, the data processing energyP is:
(e.g. Intel XScale)P=P
P-R-D Video Encoder Design
16
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 31
Step 3: optimization. We find the best configuration of complexity control parameters to minimize the coding distortion:
P-R-D Modeling
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 32
Step 3: optimization. We find the best configuration of complexity control parameters to minimize the coding distortion:
Step 4: The minimum solution gives the P-R-D function D(R, P).
He Z. et al, IEEE Transactions on Circuits and Systems for Video Technology, May, 2005.
P-R-D Modeling
[VCIP2004, IEEE TCSVT2005]
17
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 33
P-R-D Modeling
Foreman CIF Football CIF
P-R-D
OptimumControlParameters
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 34
Power-Rate-Distortion Control
1. The P-R-D analysis is offline.2. Not feasible for real-time video system control
Two Approaches for P-R-D Control
1. Statistical learning / pattern recognition approach2. Nonlinear system control approach
18
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 35
Power-Rate-Distortion Control – Approach A
P-R-D Control Profiles
Assumption
Videos of similar characteristics have similar P-R-D control profiles
Classify video segments into clusters, each with similar P-R-D control profiles.
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 36
Power-Rate-Distortion Control – Approach A
A Co-Clustering Problem
1. This is a two-way clustering problem.2. Using bi-partite graph co-clustering.
Video Cluster 1
VideoSegments
P-R-DControl Profiles
Video Cluster 2 Video Cluster 3
Once the clusters are identified, cluster average P-R-D control profile is used for control of new input videos.
19
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 37
Power-Rate-Distortion Control – Approach B
Plant(Energy-ScalableVideo Encoder)
Controller(P-R-D Control) …
ComplexityControl
Parameters
Model(P-R-D Model)
Input Video Statistics
P
Nonlinear System Identification and Control
Control parameters
State variables: distortion and power
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 38
Power-Rate-Distortion Optimization
Trade-off between Computation and Communication
Compression / Computation
Communication
For the same video quality D:
Ps R Pt
Ps R Pt R
Communication Energy
ComputationEnergy
20
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 39
Power-Rate-Distortion Optimization
Trade-off between Computation and Communication
Minimum coding distortion under energy (compression + transmission) constraint:
[He, et al, IEEE TCSVT 2006]
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 40
EncoderWirelessChannel
DecoderR [Pt, S(t)]
Power-Rate-Distortion Optimization
Trade-off between Rate, Power, and Delay[VCIP 2005]
21
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 41
Power-Rate-Distortion Optimization
Trading Bits for Joules (Energy)
γλσ nnPRnnn PRD −= 2),( 2
P-R-D Model
Minimize the overall encoding energy:
1. Energy is coupled with bits2. Energy minimization using optimum bit allocation
Observations
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 42
• For one video segment, the energy saving is impossible!
• However, for non-stationary videos, the energy saving is possible by trading bits for energy among video segments!
V1 V2 V3 V4
Time
Video
R1P1
R2P2
R3P3
……… RnPn
BitsPower
Power-Rate-Distortion OptimizationTrading Bits for Joules (Energy)
22
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 43
Power-Rate-Distortion OptimizationTrading Bits for Joules (Energy)
It can proved that energy can be saving by trading bits for joules.
Some videos are bit-sensitive while others are energy-sensitive
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 44
System Energy Consumption Characterization and Joint Application and Hardware Layers Optimization
USB
RS-2
32
USB
Power-Rate-Distortion Optimization
1. Memory access2. Disk access3. CPU energy4. Others.
23
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 45
1/25/2007 6:24:03 PM ADC
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
18:24:03.0
18:26:08.0
18:28:13.0
18:30:18.0
18:32:23.0
18:34:28.0
18:36:33.0
18:38:38.0
18:40:43.0
18:42:48.0
18:44:53.0
18:46:58.0
18:49:03.0
18:51:08.0
18:53:13.0
18:55:18.0
18:57:23.0
18:59:28.0
19:01:33.0
19:03:38.0
19:05:43.0
19:07:48.0
Current Work
Power-Rate-Distortion Analysis for Energy-Aware Video Coding
Current Drain(mA) at 5V.
XBow Stargate
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 46
Video Encoder
Processing Buffer Transmission Buffer
Effective Capacity Analysis
Encoder P-R-D Model Transmission Distortion Model
End-to-end Distortion
WirelessChannel
Scheduler
ApplicationLayer
LinkLayer
PhysicalLayer
Power-Rate-Distortion Optimization
Cross-Layer Energy Minimization
24
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
Page 47
Conclusion
1. Energy minimization of has become an important research task for video communication over portable devices.
2. Extending R-D analysis to P-R-D analysis gives us one extra dimension of flexibility in energy minimization to achieve significant energy saving gain.
3. A set of issues need to be carefully addressed at various layers of the portable video communication system.
Dept. of Electrical and Computer EngineeringUniversity of Missouri, Columbia, MO
