Real-Time Networking:Real-Time Networking:Data, Voice, and VideoData, Voice, and Video
Dan Schonfeld
Multimedia Communications Laboratory
ECE Department
University of Illinois
Chicago, Illinois
Multimedia Communications Multimedia Communications LaboratoryLaboratory
Rashid AnsariRashid AnsariAshfaq KhokharAshfaq KhokharDan SchonfeldDan Schonfeld
Oliver YuOliver Yu
A Vision To The Future!A Vision To The Future!Real-Time High-Quality Global Video Communications over High-Real-Time High-Quality Global Video Communications over High-
Speed NetworksSpeed Networks
Research AreasResearch Areas
Video NetworkingVideo TrackingVideo Retrieval
Video NetworkingVideo Networking
Collaborators:Collaborators:Rashid Ansari (UIC)Rashid Ansari (UIC)Bulent Cavusoglu (UIC)Bulent Cavusoglu (UIC)Tom DeFanti (UIC)Tom DeFanti (UIC)Jason Leigh (UIC)Jason Leigh (UIC)Emir Mulabegovic (UIC)Emir Mulabegovic (UIC)Oliver Yu (UIC)Oliver Yu (UIC)
Homeland Security Homeland Security Application TestbedApplication Testbed
Chicago Video Surveillance Monitoring of Biochemical Sensor Arrays Helmet Mounted Cameras and Portable Sensors
for Search/Rescue Operations Dedicated Emergency Telephony Network
– Public network lack of reliability during emergency
Real-Time Real-Time Surveillance/MonitoringSurveillance/Monitoring
Central Monitoring– Ok as long as links are
dedicated
Central Monitoring
Search and rescue
Remote sharing with agencies in the field
Mobile and Portable Emergency Response Center
Next Generation Internet Next Generation Internet ArchitectureArchitecture
CR-LDPBGPIGP
ICMPRSVP
Signaling Protocol
Out-of-Band Associated
Signaling Transport
Traffic Control
Fiber/DWDM
SONETATM/PPP/HDLC
IP Routing
Path SelectionMPLS Forwarding
IP QoS Control
HTTP/SMTP/FTP/TELNET
Real-time Multimedia
Middleware
RTPUDP
TCP
Best Effort Guaranteed & Controlled-Load
Non-Real-time Real-time
IP / MPLS
Topology Distribution
Transport
User Application
Current
Internet NGI
Extension
Net
wor
k &
Lay
er
Man
agem
ent
VIPER: FEC/UDP Over VIPER: FEC/UDP Over STARTAPSTARTAP
(Chicago-Amsterdam)
Latency of transmitting 100 packets underUDP, TCP, FEC/UDP with 3:1 redundancy.
0
50
100
150
200
250
300
350
400
0 500 1000 1500 2000 2500
Packet size in bytes
1-way latency in ms
UDP
TCP
FEC over UDP
APPLICATION
RTP
UDP
IP
DATA LINK
PHYSICAL
TRANSPORT
Picture Type
f_code
MBZ T TR AN N S B E P FBV BFC FFV FFC
MPEG-1 Header extension
X Ef(0,0)
f(0,1) f(1,0) f(1,1) DC PS T P C Q V A R H G D
MPEG-2 Header extension
Real-Time Real-Time Transport Transport Protocol Protocol
(RTP)(RTP)
AdaptiveAdaptiveFECFEC
RTP MEDIAPACKETS
ADAPTIVEFEC
ENCODER
RTP FECPACKETS
MPEG-2 RTP NETWORK RECEIVER
1 2 3 4 5 6 7 8 9 1030
32
34
36
38
40
42
44
46
48
50
Packet Loss Ratio Percentage
PSNR in dB
AFECstaticIPBstaticFECoptimalFEC
(c) AFEC
(b) Static IPB(a) Static FEC
(d) Original
FEC SimulationsFEC Simulations
DiffServDiffServ
HOST
EDGE ROUTER CORE ROUTER CORE ROUTER EDGE ROUTER
HOST
TOKEN BUCKET
WEIGHTED FAIRQUEUING
(WFQ)
CHECK IF THERE IS ANYTOKENS AVAILABLE FOR
THE ASSIGNED DSCP
YES
ASSIGN NEXT AVAILABLEDSCP FROM THE TABLE
TOKENS
NO DSCP TABLEAF31AF32AF33AF21AF22AF23AF11AF12AF13
BEST EFFORT
QUEUESAF3
AF2
BESTEFFORT
RANDOM EARLY DETECTION(RED)
TRANSMITTED PACKETS
DROPPED PACKETS
MARKERWEIGHTS
FOR PACKETS
HOSTHOST
HOST
OTHER SOURCES THATCONTRIBUTES TRAFFIC
VIDEOSOURCES
VIDEO SOURCE WITH THEPROPOSED MARKING
ALGORITHM
RANDOM EARLY DETECTION(RED)
RANDOM EARLY DETECTION(RED)
5.7 5.75 5.8 5.85 5.9 5.95 6 6.05 6.1 6.1518
19
20
21
22
23
24
25
26
27
28
Mbps
PSNR in dB
MotionIPBGreedyOptimum
(c) Motion
(b) IPB(a) Greedy
(d) Original
DiffServ SimulationsDiffServ Simulations
Rate-ControllerRate-Controller
NETWORK
DROPPEDPACKETS
HOST HOST
HOST
HOST
HOST
OTHER FLOWS
HOST
HOST
HOST
OTHER FLOWS
RATE CONTROLLED FLOW
( )ip μ
LINEAR STATEFEEDBACKREGULATOR
$
$[ ]
( )
( ),
min( )i
tot
D
E Dp µ
p µ
$( )p µ
Nack packetsor RTCP report
( )iμ
µ
Networkpk
KalmanPredictor
vk
zk
wk
z-1 inv()
1kμ +
( ) 1kμ
−
kα inv()
1kα−
1
−
+
1( )k kp μ+)
11 kα−−
uk
uk
11(1 ) ( )k k kpα μ−+− )
−
+kp
z-1( )k kp μ)
1( )kp + µ)1 1
1
( )
( )
dk k
k
choosep μ
μ+ +
+ ∈µ
)
1kμ +
( )k
Generate
u µ
5
1
2
7
2
5
2
2 4
4
4
7 5
4
6
(b) Choke-ORCA(a) Choke-Only
Rate-Control SimulationsRate-Control Simulations
Lightweight Streaming Lightweight Streaming Protocol (LSP)Protocol (LSP)
Videofile
Framer Discriminator Packetizer Packetbuffer
Sender
ReceiverControlShared
parametersand statistics
RTX
To client
Control messages from theclient
Nominal frame rate (NFR)
Actual frame rate (AFR)
Server architecture
SEQ1 SEQ2(lost) SEQ3 SEQ4(lost) SEQ5 SEQ2 (rtx) SEQ4(rtx) NACK2 (ignored)
NACK2 (NACK2 + NACK4)
Receiver Sender
PacketsTrans-mitted
Packets received
UDP LSP LSP-PMN
DSL 94% 99% 100%
Wi-Fi 95% 96% 98%
LSPLSP
PSNR = 18 dBms PSNR = 33 dBms
QoS Control QoS Control Over Wireless & Over Wireless &
Core NetworkCore Network
CDMA-Based CDMA-Based Admission & Admission & SchedulingScheduling
(Oliver Yu)(Oliver Yu)
Internet
Backbone
Gateway
Backbone
Wireless QoS Control
Core QoS Control
Internet QoS Control
Router
MSC
AP-CAC
Backbone
BS
FDWFQ-MAC
AMDG-CAC
Wireless MAC ProtocolsWireless MAC Protocols(Khokhar)(Khokhar)
Listen Idle
Zzzz
Zzzz
Zzzz
Zzzz
Zzzz
Zzzz
Zzzz
Zzzz
Zzz
z
Zzzz
Duty Cycle
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
Event Arrival Rate
Normalized Power Consumption
TDMA-W, 50 nodesTDMA-W, 100 nodesTDMA-W, 200 nodes10% S-MAC, 50 nodes10% S-MAC, 100 nodes10% S-MAC,200 nodes
9.30%
8.16%
5.13%
0.56%
Channel
Call AdmissionController
Slot Level Scheduler
X
X
Center for Global Multimedia Center for Global Multimedia Mobile CommunicationsMobile Communications
Internet over CableInternet over Cable
Digital Subscriber LinesDigital Subscriber Lines
A typical ADSL equipment configuration.
Low-Earth Orbit SatellitesLow-Earth Orbit SatellitesIridiumIridium
(a) The Iridium satellites form six necklaces around the earth.
(b) 1628 moving cells cover the earth.
GlobalstarGlobalstar
(a) Relaying in space.(b) Relaying on the ground.
The 802.16 Physical LayerThe 802.16 Physical Layer
The 802.16 transmission environment.
Computer Network Computer Network InfrastructureInfrastructure
UICNU
NCSA
UCSD
Optical Networking Transport Encoding
& Protocols Wired and Wireless
Network Integration Circuit and Packet Switched
Network Deployment
Wireless CommunicationsWireless Communications
Router Wireless Access Pointoptical Switch Wireless Terminal
Optical Core
PacketData Network
Wireless Access Network:• IEEE 802.11 (Year 1)• IEEE 802.16 (Years 2 & 3)• IEEE 802.20 (Optional)
Wireless AccessNetwork
Opportunistic Resource Allocation & Admission Control
Channel Estimation Power-Efficient
Wireless Protocols High-Capacity
Wireless Networks
Applications & PrototypesApplications & Prototypes
Video Communications Tele-Education Natural Event Monitoring Geosciences Monitoring Environmental Assessment Emergency Management Elderly Care Medical Diagnosis Remote Robotic Surgery
Visualization & DevicesVisualization & Devices
High-Resolution Scalable Displays
High-Resolution Capture
Interactive Tools
Intelligence SharingIntelligence Sharing
Real Time Monitoring and Real Time Multimedia Retrieval and Sharing across the continent
Video TrackingVideo Tracking
Collaborators:Collaborators:Nidhal Bouaynaya (UIC)Nidhal Bouaynaya (UIC)Karthik Hariharakrishnan (Motorola Research)Karthik Hariharakrishnan (Motorola Research)Dan Lelescu (NTT DoCoMo Research)Dan Lelescu (NTT DoCoMo Research)Josh Meir (NeoMagic)Josh Meir (NeoMagic)Magdi Mohamed (Motorola Research)Magdi Mohamed (Motorola Research)Wei Qu (UIC)Wei Qu (UIC)Philippe Raffy (R2 Technology)Philippe Raffy (R2 Technology)Fathy Yassa (NeoMagic)Fathy Yassa (NeoMagic)
MotivationMotivation
Target Tracking Surveillance Retrieval Video Coding Video Communications Videoconferencing Virtual Reality Human-Computer Interaction Computer Animation
VORTEXVORTEX
Reference frame
Object cluster
VORTEX: Video Retrieval and Tracking from Compressed Multimedia Databases
Template Template Matching [sec] VORTEX [sec]
Object #1 45.22 0.0084
Object #2 39.36 0.0092
VORTEXVORTEX
Adaptive Block Matching Adaptive Block Matching (ABM)(ABM)
Method Time [sec]
ABM 10
Partition Projection 165
Partition Lattice Operators 193
MBPFCondensation filter
Motion-Based Motion-Based Particle FiltersParticle Filters
Multi-Object Particle FiltersMulti-Object Particle Filters
. . .. . .
. . .
......
...
......
...
21x
1
mx 2mx
22x
1
2x1
1x 1
tx
2tx
m
tx
1
1z
2
1z
1mz 2
mz
2
2z
1
2z 1
tz
2
tz
mtz
HMM
MHMM
x1 x2 xt. . . .
z1 ztz2
p(zt|xt)
p(xt|xt-1)
Experimental ResultsExperimental Results
The Dynamic Graphic Model for Multiple Interactive Objects In Two Frames
IDMOTIDMOT
Magnetic-Inertia ModelMagnetic-Inertia Model
Reward
Punish
Video Tracking and Video Tracking and FoveationFoveation
(Ansari & Khokhar)(Ansari & Khokhar)
Future Research:Future Research:Video TrackingVideo Tracking
Randomly Perturbed Active Surfaces Video Stabilization Auto-Focus Recovery Pose Estimation and Feature Tracking Video Animation Stereography from a Single Camera Multiple Camera Mosaics Multiple Camera Tracking Low-Power Particle Filters
Video RetrievalVideo Retrieval
Collaborators:Collaborators:Faisal Bashir (UIC)Faisal Bashir (UIC)Ashfaq Khokhar (UIC)Ashfaq Khokhar (UIC)Dan Lelescu (NTT DoCoMo Research)Dan Lelescu (NTT DoCoMo Research)Fatih Porikli (Mitsubishi Research)Fatih Porikli (Mitsubishi Research)
MotivationMotivation
• Video Surveillance• Sign Language Recognition• Sports Video Analysis• Animal Mobility Experiments• Moving Object Databases• Video and Sensor Databases
Spectral ClusteringSpectral Clustering
Trajectory RetrievalTrajectory Retrieval
0
500
1000
1500
2000
2500
3000
1 2 3 4 5 6
PCA-Global
PCA-Seg Euc
PCA-Seg Str
Lei Chen
Gaussian Mixture ModelsGaussian Mixture Models
Figure: 1-Sigma contours of GMM’s learnt from three classes.(a) ‘Norway’. (b) ‘Alive’. (c) ‘Crazy’.
1
cN
i i ii
P( y ) ( y; , )π μ=
Θ = ∑∑ ¥
HMM for Class NHMM for Class 1
ClassificationClassification
…
…
Gaussian Mixtures
Training Set Database
Classification:
[ ]( )1
1m i
i , ,Larg max p Y , ,Y λ
∈ LL
AccuracyAccuracy
Datasets
ASL
HJSL#Classes : # Trajectories
2:138 4:276 8:552 16:1104 29:2001 38:2622
HMM 0.9638 0.9167 0.8587 0.7790 0.6882 0.6609 0.9074
GMM 0.9855 0.8949 0.8514 0.7455 0.6672 0.6400 0.8981
Moghaddam 0.9420 0.9312 0.8297 0.7283 0.5592 0.6175 0.4537
Accuracy values for various class sizes from ASL data set and the HJSL dataset (last column). Column headings are shown as (number of classes:number of trajectories) for the ASL dataset at different sizes.
Shape RepresentationShape Representation
Curvature Scale SpaceCurvature Scale Space
CSS Images of a Trajectory and its 36-degree rotated versionFigure: An example high jump trajectory and its translated, rotated and non-uniformly scaled version, along with their CSS images.
PerformancePerformance
Indexing Time (sec.)
(408 Traj.)
Retrieval Time (sec.)
(15 Traj.)
PCA Centroid 178.9270 8.2920
Hybrid PCA 175.2020 27.5400
CSS 1508.3 28.0500
Future Research:Future Research:Video RetrievalVideo Retrieval
Trajectory OcclusionCamera MotionMultiple CamerasMultiple TrajectoriesVideo MiningJoint Retrieval, Recognition, & MiningMulti-Modality Feature Integration