04/08/2304/08/23 uri="http://www.globalmmcs.org" email="uri="http://www.globalmmcs.org" email="[email protected]"[email protected]"
11
Design and Implementation of Audio/Video Collaboration System
Based on Publish/subscribe Event Middleware
PTLIU Laboratory for Community GridsGeoffrey Fox, Wenjun Wu, Ahmet Uyar, Hasan Bulut
Computer Science DepartmentIndiana University, Bloomington IN 47404
http://www.globalmmcs.org
CTS04 San Diego 19 January 2004
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 22
A/V Collaboration Systems H323
H.323 is defined as an umbrella standard specifying the components to be used within an H.323-based environment.
SIPThe Session Initiation Protocol (SIP) defines how to establish, maintain and terminate Internet sessions including multimedia conferences
Access Grid enhanced Mbone A/V tools ( VIC, RAT )
Internet 2 network ( Multicast support )
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 33
XGSP Collaboration Framework To integrate heterogeneous systems into one
collaboration system• A unified, scalable, robust “overlay” network is needed to
support A/V and data group communication over heterogeneous networking environments.
• A common A/V signaling protocol has to be designed to support interactions between different A/V collaboration endpoints.
• Different A/V endpoints should collaborate in the same collaboration session.
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 44
Group Communication in Collaboration Collaboration applications usually need a group
communication service for both multipoint data and control information delivery• Centralized conferencing systems usually depend upon a
single conference server
• Distributed conferencing systems use IP multicast
Access Grid uses Internet2 multicast for audio/video transmission.
• Problems:
Centralized conferencing systems don’t have good scalability
IP multicast has not become ubiquitously available
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 55
Narada Broker Network
Database
Resource
Broker
Broker
Broker
Broker
Broker
Broker
Software multicast
(P2P) Community
(P2P) Community
For message/events service(P2P) Community
(P2P) Community
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 66
Narada Brokering Narada Brokering from the Community Grid Labs is
adopted as a general event brokering middleware• supports publish-subscribe messaging model with a dynamic
collection of brokers
• provides services for TCP, UDP, Multicast, SSL and raw RTP clients
• Supports the communication through firewalls and proxies
• operate either in a client-server mode like JMS or in a completely distributed JXTA-like peer-to-peer mode.
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 77
Narada Brokering
Covers the heterogeneity of network transportation and provides unified multipoint transportation API• Software multicast • Communication over firewalls and proxy boundaries • Communication over multiple transports • Application level Quality of Service
Filter messages to slow (collaborative/real-time) clients Provides robust, scalable and high efficient multipoint
transportation services • Availability and scalability • Efficient routing and bandwidth utilizations
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 88
A/V Collaboration over publish/subscribe Middleware video streams (VS) VS {v1, v2, … vm }
audio streams (AS) AS { a1, a2, … an }
A/V endpoints: E1, E2, … En
Each endpoint may send a single or multiple video streams, but only send an audio stream
Different types of A/V endpoints have different collaboration capabilities. • Multicast endpoints are able to receive multiple video and audio
streams, display all the video streams in their screens, and mix all the audio streams by themselves
• Unicast endpoints like Polycom Via Video can only receive and play a single video and audio stream
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 99
Publish/subscribe of A/V A stream in VS and AS is regarded as a “topic” Each RTP packet from this stream is regarded as an
“event” for this topic Only the sender of this stream can “publish” A/V
events to this topic Other endpoints need to subscribe to this topic in order
to receive the stream data Create mixed video and audio stream topics for unicast
endpoints
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1010
Major Events for each stream topicEvent Name Change in the stream
NewStreamEvent This stream has been created
ByeEvent The stream gives a BYE RTCP packet, indicating it has left the session
TimeOutEvent This stream has not send any RTCP packet for a long time, indicating it may have left the session
Active-To-Passive The stream has stopped sending RTP packet
Passive-To-Active The stream resumed sending RTP packet
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1111
A/V Collaboration Architecture
NaradaBrokeringPublish/Subscribe Messaging
AVSessionServer
SIPGateway
H.323Gateway
H.323Client
SIPClient
AccessGrid
Multicast
XGSPWeb
Server
AudioMixer
VideoMixer
ImageGrabber
Media ServersAudioMixer
VideoMixer
ImageGrabber
RTPLink
RTPLinkRTPLink
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1212
RTP packets encapsulation RTPLink: RTP Packets ~ Narada Brokering Event Every legacy A/V client needs one corresponding
RTPLink to be set up at a broker within broker network.
Unicast RTPLink: Integer Topic Numbers for RTP and RTCP
Multicast RTPLink:
A reflector between NaradaBrokers and multicast groups, encapsulating raw RTP packets from a multicast IP address to RTP events, publishing these events to NaradaBrokers, and forwarding the data it receives from broker network on the same IP address
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1313
Audio Mixer, Video Mixer, Image Grabber Audio Mixing
The audio mixer creates a mixed audio stream from all the audio streams in the session
Video Mixing Video mixing makes the unicast users watch the pictures of
multiple participants in a meeting through one video stream Video Thumbnail visualize the VS set in the session, embedded into the control panel
of each endpoint, which Image grabbers capture video streams and save them as static JPEG
files.
All the media processing components can be distributed among the pool of the media servers connected to NaradaBrokering infrastructures.
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1414
H.323, SIP Gateway Servers, A/V Session Server H.323 and SIP gateway transform these protocol
specific messages into XGSP signaling messages so that H.323 and SIP A/V endpoints could communicate with the XGSP A/V session server
The session server implements session management logics • creating/destroying A/V sessions
• allowing endpoints to join/leave session
• Allowing users to make audio/video selection, managing A/V application components
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1515
Implementation Java Media Framework (JMF) : Openh323 is basis of H323 Gateway NIST SIP stack is basis of SIP Gateway NaradaBrokering is open source messaging from
Indiana
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1616
Demo Pics ( I )
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1717
Demo Pics ( II )
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="[email protected]"uri="http://www.globalmmcs.org" email="[email protected]" 1818
Demo Pics ( III )
04/08/2304/08/23 uri="http://www.globalmmcs.org" email="uri="http://www.globalmmcs.org" email="[email protected]"[email protected]"
1919
Thank you!
Please watch our demo at 5:30 PM !