Using Dialogic® Media Toolkit API for Image Overlay on Video ...

Using Dialogic® Media Toolkit API for Image Overlay on Video Streams

Small Logo

Medium Logo

Large Logo

Application Note

Executive SummaryDialogic® Multimedia Platform Software for AdvancedTCA Release 2.0 and Dialogic® Multimedia Kit Software Release

1.0 for PCIe support the ability to overlay images on video streams using the image overlay feature implemented in

the Dialogic® Media Toolkit API library. This application note demonstrates how the image overlay feature (called the

“MTK API” in this application note) can be used by an application so that a user can play an image (YUV or JPEG)

over streaming video. Another application note discusses using an open source project for converting text into a

graphics image, which can then be used by the image overlay feature.

Application NoteUsing Dialogic® Media Toolkit API for Image Overlay on Video Streams

Table of ContentsIntroduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Application Use Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Video Conference Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

About Dialogic® Media Toolkit API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

MTK API Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Before Using the MTK API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

MTK API Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Creating and Applying Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Applying the Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Bounding Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Modifying the Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Other Functionalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Timers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Cleaning Handles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Demo Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Building the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Starting the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Determining Image Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Appendix A: Dialogic® Media Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Interdevice Connnections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

For More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Application NoteUsing Dialogic® Media Toolkit API for Image Overlay on Video Streams

1

Introduction

Dialogic® Multimedia Platform Software for AdvancedTCA Release 2.0 (MMP Software 2.0 for ATCA) and Dialogic® Multimedia Kit Software Release 1.0 for PCIe (MMK Software 1.0 for PCIe) support the ability to overlay images on video streams by using the image overlay feature provided by the Dialogic® Media Toolkit API library.

This application note demonstrates how the MTK API can be used by an application to manipulate video streams. Various application use cases involving image overlay are provided, and sample applications that demonstrate this functionality are provided for download (see the For More Information section).

Application Use Cases

The following are a few examples of application use cases for using image overlays:

Applying a company logo to a video stream (single video • stream or a conference)

Applying conferee name, title, or a conference title•

Displaying time left notifications, for example, “5 minutes • left in conference”

Displaying video clip name or clip owner information•

Displaying stream control icons for Play, Pause, Fast-• Forward, etc.

Displaying call waiting notifications•

Some of the use cases listed above involve using text, and other use cases use images as overlays on a video stream. This application note focuses on image overlays on video streams. The text-to-image conversion is discussed in another application note, Using Dialogic® Media Toolkit API and ImageMagick for Image Overlay on Video Streams (see the For More Information section).

Video Conference Examples

Figures 1 through 4 show some examples of image overlay on a video conference. Participants in this video conference are SIP peers connected to ipmB1C1 and ipmB1C2. Additionally, a 3G terminal is participating in the conference via the m3gB1C1 device. It is also possible to have a multimedia device in the video conference, such as for the purposes of recording the video conference or playing some media into the video conference, represented as mmB1C1 in the figures.

Image overlays in this example video conference can be applied in a way so that some or all participants see the overlays via the direction parameter. This parameter will be discussed in the Overlay Direction section.


Application Note

2

m3gB1C1

Conference

mmB1C1 ipmB1C2

eSM_OVERLAY_DIRECTION_DEVICE Overlay to apply

ipmB1C1

Input to Video Conference

Output from Video Conference

Video Conference Output Received by Device’s Network Peer

Input to Device from Network Peer

Figure 1. All Participants See the Overlay for ipmB1C1 Participant

Figure 2. Overlay Only Seen by ipmB1C1 Participant

Figure 3. Each Participant Sees a Different Overlay


Application Note

3

m3gB1C1

Conference

mmB1C1 ipmB1C2

eSM_OVERLAY_DIRECTION_NETWORK

Overlay to apply

ipmB1C1





ConferenceeSM_OVERLAY_DIRECTION_NETWORK

Overlay to apply

ipmB1C1


ipmB1C2


m3gB1C1


mmB1C1





About Dialogic® Media Took Kit API

Advanced video features of MMP Software 2.0 for ATCA and MMK Software 1.0 for PCIe include image overlays and video conference party layouts. The Dialogic Media Toolkit API library’s image overlay feature (the Dialogic Media Toolkit API is called the “MTK API” in this application note) provides those features. The Dialogic Media Toolkit API library consists of general-purpose structures and attribute templates, as well as API functions for building and manipulating media-related items, such as video layouts and bitmaps. The MTK API allows manipulation of an incoming media stream to a device or outgoing media stream from a device - that is, it allows a user to place an image (YUV or JPEG) over a streaming video. The user creates a template of the image and defines the area on the video screen in which the image will be displayed. More information about the MTK API can be obtained from the API reference, Dialogic® Media Toolkit API: Library Reference (see the For More Information section).

Image overlay operations are applicable to MM, IPM, and M3G devices, each of which has audio and video ports that are either internal device connection ports (evolution of the CT-Bus) or external network ports (such as audio and video RTP streams of IPM devices). See Appendix A: Dialogic® Media Devices section for an overview of various media devices and how they can be connected to each other and/or to the external network.

MTK API Family

The MTK API family is a collection of APIs that allow an application to manipulate video streams. The APIs are categorized as follows:

• mtk_xxx() — General purpose functions used in creating and destroying image, file, bitmap, and frame templates. These templates are then used by other APIs in the MTK API family. This category also contains functions that perform general error handling functionality.

• lb_xxx() — Layout builder functions. These functions are used for creating video conference layouts. This API is not discussed in this application note.

• sm_xxx() — Stream manipulation functions. These functions allow overlays to be applied to a stream. The stream could be coming into a device or going out from it.

• ob_xxx() — Overlay builder functions. These functions, in conjunction with the mtk_xxx functions are used for creating and manipulating image overlay templates.


Application Note

4

ConferenceeSM_OVERLAY_DIRECTION_NETWORK

Overlay to apply

ipmB1C1


ipmB1C2


m3gB1C1


mmB1C1





Figure 4. All Participants See the Same Overlay

Before Using the MTK API

Before an overlay can be applied, the following steps would need to performed to set the application variables.

1. Determine the image attributes. The image attributes are: image type and image dimensions. The Dialogic MTK API does not have functions that help determine these attributes. See the Determining Image Attributes section for one method of many that aid in determining image attributes. The MMP Software 2.0 for ATCA firmware supports YUV and JPEG image types.

2. Determine image scaling. An image can be applied at 100% scale (no scaling) or some other scaling. For better quality, the image should not be scaled. If the image is scaled higher or lower, the aspect ratio of the image should be retained.

3. Determine the position within a frame of a video stream where the image should be overlaid. The position can be either a percentage of a video stream’s frame dimensions or an absolute position. The upper left-hand corner of the frame is location (0,0). Figure 5 shows a frame and the coordinate system orientation. The overlay position can also be specified as a horizontal and vertical justification. Therefore, the application does not need to calculate position locations; it can use the API to specify the position justification as Left, Right, Top, Bottom, or Center for both horizontal and vertical settings. See the Bounding Frame Justification section for examples.

If any attributes are to be applied in terms of absolute coordinates, the application needs to separately track the video stream resolution (QCIF or CIF).

The application must have connected devices together using the dev_PortConnect(DMFL_TRANSCODE_ON). If a device is connected to another resource using the dev_Connect() function or dev_PortConnect(DMFL_TRANSCODE_NATIVE), the overlay operation cannot be performed.

Figure 5. Coordinate System

MTK API Concepts

The MTK API is not bound to a Dialogic® device initially. The application first uses the MTK API to obtain various handles and then eventually, the media is applied to a Dialogic device.

Specifically, before an overlay can be applied to a media stream, applications must create two types of templates: overlay templates and bounding frame templates. These templates aid in associating the image data and placement of the image on a media stream.


Application Note

5

(0,0)

(0,M) (N,M)

(N,0)

The overlay template initialization activity associates an image or image buffer to some handles, sets the fill style of the image in an area of the stream that will contain the image, and sets the overlay visibility duration once the overlay is active on a media stream.

The bounding frame template represents the section of the video frame in which the overlay appears in the media stream.

Creation of the overlay and bounding frame templates are shown in the upper-left and upper-right rectangles of Figures 6 and 7, respectively.

Creating and Applying Overlays

An application may use a file as the source of an image, or it may load the image contents into a buffer and use that image buffer. Figure 6 and Figure 7 show the four steps required for creating an overlay and adding the overlay onto a device’s media stream:

1. Create one or more overlay template(s).

2. Create one or more bounding frame template(s).

3. Bind an overlay template to a bounding frame.

4. Apply an overlay template to a device.

Steps 1, 2, and 3 can be done in advance of when the overlay

is needed to be applied to a device. The figures show N overlay

and M bounding frame templates being created in advance.

Thus, different overlay schemes can be created in advance

and particular templates can be applied during application

runtime, as is done by Step 4.

One or two overlays may be applied to a media device in a

single sm_AddOverlays() function call. A device can support a

maximum of two overlays at a given time.


Application Note

6

sm_AddOverlays(SRL_DEVICE,{direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2)

4. Apply up to 2 template(s) to media device

hBoundingSnapshot = ob_SetOverlayBoundingFrame(hOvlTemplate,boundingFrameTemplate)

3. Bind the overlay templates to bounding frames

ob_SetOverlayDuration(hOvlTemplate,...)

ob_SetOverlayFillStyle(hOvlTemplate)

hOvlTemplate = ob_CreateImageOverlayTemplate(hMedia)

hMedia = mtk_CreateMediaFileTemplate(hImageTemplate,

<file name>)

hImageTemplate = mtk_CreateImageTemplate(eMTK_IMAGE_FORMAT_JPEG)

1. Create N overlay templates

mtk_SetFramePosition(boundingFrameTemplate,...)

mtk_SetFrameSize(boundingFrameTemplate,...)

hBoundingFrameTemplate = mtk_CreateFrameTemplate()

2. Create M bounding frame templates (M)

Figure 6. Adding JPEG Image Overlay

Figure 7. Adding YUV Image Overlay

Applying the Overlays

The same overlay template can be used to apply the same overlay to different media devices. Figure 8 shows that the SMEV_

ADD_OVERLAY event, returned to the application for Device A, contains hDeviceASnapshot. This snapshot handle is a reference

to the parameters in the overlay template when the overlay is added to the given device. Any modification of the overlay or deletion

of the overlay must be done using this device-specific snapshot handle. For the same reason, changes to the original overlay

template are not made to the added overlay on the device.

Note: If a device is connected to another device using dev_Connect() function or dev_PortConnect(DMFL_TRANSCODE_NATIVE),

the overlay cannot be applied. In this instance, the dev_PortConnect(DMFL_TRANSCODE_ON) must be used prior to applying

the overlay.


Application Note

7

4. Apply up to 2 template(s) to media device

3. Bind the overlay templates to bounding frames

1. Create N overlay templates 2. Create M bounding frame templates (M)

sm_AddOverlays(SRL_DEVICE,{direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2)

hBoundingSnapshot = ob_SetOverlayBoundingFrame(hOvlTemplate,boundingFrameTemplate)

ob_SetOverlayDuration(hOvlTemplate,...)

ob_SetOverlayFillStyle(hOvlTemplate)

mtk_SetYUImageFormat(hImageTemplate,eMTK_YUV_IMAGE_FORMAT_420)

hImageTemplate = mtk_CreateImageTemplate(eMTK_IMAGE_FORMAT_YUV)

mtk_SetFramePosition(boundingFrameTemplate,...)

mtk_SetFrameSize(boundingFrameTemplate,...)

hBoundingFrameTemplate = mtk_CreateFrameTemplate()

hMedia = mtk_CreateMediaFileTemplate(hImageTemplate,

<file name>)

hOvlTemplate = ob_CreateImageOverlayTemplate(hMedia)

Figure 8. Applying Overlay

Overlay Duration

An overlay can have a finite duration or infinite duration on each media stream generated by the associated device. The ob_SetOverlayDuration() function is used to either specify a duration in units of milliseconds for finite durations or MTK_INFINITE for infinite durations.

Finite Duration

If an overlay has a finite duration, the overlay will be overlaid from the beginning of any stream generated by the device until the specified time has elapsed, at which time it will disappear from the stream. Note that the application is not notified of an overlay expiry. After the expiry, the overlay is no longer visible on the media stream. The application must remove the overlay if the user does not wish for the overlay to be seen in subsequent streams generated by the device.

Overlays with finite duration may help in creating animation effects. This is discussed in the Animation section.

Infinite Duration

An overlay can have an infinite duration. The overlay will be visible on each media stream generated by the device for the full duration of each stream until the application removes it from the device.

Overlay Lifetime

An overlay applied on a device is not removed from that device until the sm_RemoveOverlays() function is called. If an overlay has a finite duration, the sm_RemoveOverlays() must still be called by the application as discussed in the previous Overlay Duration section.


Application Note

8

sm_AddOverlays(hDeviceA,{direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2)

hDeviceASnapshot1 ... hDeviceASnapshot2

SMEV_ADD_OVERLAY

SMEV_ADD_OVERLAY

SMEV_ADD_OVERLAY

sm_AddOverlays(hDeviceB,{direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2)

hDeviceBSnapshot1 ... hDeviceBSnapshot2

sm_AddOverlays(hDeviceC,{direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2)

hDeviceCSnapshot1 ... hDeviceCSnapshot2

Device A Device B Device C

If an application closes a device without removing the overlay, the overlay is still associated with that device in the firmware. If a new application process is started, the overlay will still be present on that device. The same is true if the previous application process exited abnormally; the overlay will still be present on the same device when it is opened in a new application process. The sm_RemoveAllOverlays() function is provided for such scenarios. The application should call this API at the start of a new process to ensure that no overlays from a previous application are still visible. If the firmware is restarted, overlays will be removed. See Figure 9 for a diagram of this discussion.

Figure 9. Overlay Lifetime

Overlay Direction

Overlay direction allows an overlay to be visible on either the outgoing or incoming media stream of a device. Two different directions are defined:

• eSM_OVERLAY_DIRECTION_DEVICE — The overlay is placed on the data stream between the device in question and any other device with which it is connected.

• eSM_OVERLAY_DIRECTION_NETWORK — The overlay is placed on the data stream between the device in question and the external network.

For a media device that does not normally have a network interface, such as a multimedia device, the external network is considered the sink (that part of the system that receives information) associated with that device, such as a recorded file. Therefore, if a mm_Record(hMMDevice, …) operation is pending, applying an overlay to this multimedia device handle with direction as eSM_OVERLAY_DIRECTION_NETWORK will result in a recorded file with the overlay. If mm_Play(hMMDevice,…)


Application Note

9

Application Process Exits

Device A

Application Process Starts

Add Overlay

Close Device

...

New Application Process

sm_RemoveAllOverlays()

was issued and an overlay applied with direction eSM_OVERLAY_DIRECTION_DEVICE, another device that is listening to this multimedia device will receive a media stream with an overlay.

Removing Overlays

The same overlay can be applied to different media devices. The overlay template handle used during the sm_AddOverlays() function is not the same handle that is used to remove the overlay. When overlay(s) are added, the resultant SMEV_ADD_OVERLAY event is generated and sent to the application. This event contains an overlay snapshot handle for each overlay added to that specific device. If the application needs to remove an overlay, the snapshot handle returned in the SMEV_ADD_OVERLAY event is used in the sm_RemoveOverlays() function. See Figure 10 for an example.

Figure 10. Removing Overlays

Bounding Frames

A bounding frame is a region that specifies the placement of an image on top of the media stream. The bounding frame has a position and size and is associated to an image (or image buffer) to be overlaid.

Bounding Frame Position

The position of a bounding frame is set using mtk_SetFramePosition(MTK_FRAME_HANDLE a_hFrame, float a_x, float a_y, eMTK_POSITION_TYPE a_ePositionType) on the handle returned by the mtk_CreateFrameTemplate() function.

The meaning of a_x and a_y arguments depends on the a_ePositionType parameter.


Application Note

10

sm_RemoveOverlays(hDeviceA,hDeviceASnapshot1 ... hDeviceASnapshot2)

sm_RemoveOverlays(hDeviceB,hDeviceBSnapshot1 ... hDeviceBSnapshot2)

sm_RemoveOverlays(hDeviceC,hDeviceCSnapshot1 ... hDeviceCSnapshot2)

SMEV_REMOVE_OVERLAY

SMEV_REMOVE_OVERLAY

SMEV_ADD_OVERLAY

Device A Device B Device C

Position type can be:

eMTK_POSITION_TYPE_PIXEL• — Pixel based coordinates; (0,0) is upper, left-hand corner

eMTK_POSITION_TYPE_PERCENT• — Percentage based coordinates; (0,0) is upper, left-hand corner

• eMTK_POSITION_TYPE_JUSTIFICATION — Justification-based coordinates. See Bounding Frame Justification section for examples.

Bounding Frame Size

The size of a bounding frame is set using mtk_SetFrameSize(MTK_FRAME_HANDLE a_hFrame, float a_w, float a_h, eMTK_SIZE_TYPE a_eSizeType) on the handle returned by the mtk_CreateFrameTemplate() function.

The meaning of a_w and a_h arguments depends on the a_eSizeType parameter. The size type can be:

eMTK_SIZE_TYPE_PIXEL• — pixel-based sizing

eMTK_SIZE_TYPE_PERCENT• — percentage of the bounding frame-based sizing

The size of a bounding frame impacts the quality of the resultant overlay. The image will be stretched to fit the bounding frame. Note that the bounding frame is not visible — only the image to be overlaid is visible on top of the media stream.

Figure 11 shows examples of an image being applied to a media stream and the results based on the bounding frame size.


Application Note

11

1

2

3

Media StreamBounding FrameImageEx Overlay Result

Figure 11. Bounding Frames

• Example 1 — If the bounding frame does not have the same dimensions as the image, then the image will be stretched and may be distorted.

Example 2• — If the bounding frame is the same size as the image, then scaling distortion is not present.

• Example 3 — If the image is larger than the bounding frame, then the image will be scaled down to fit the bounding frame. This too may result in distortion. A main concern is that the proportion of the image and the frame should match closely.

Bounding Frame Justification

An application can specify the position in terms of justification instead of basing it on the base frame.

Some justification examples are shown in Figure 12.

Figure 12. Overlay Justification

Table 1 maps the example image overlay locations in Figure 13 to the justification parameters.

Image Horizontal Justification Vertical Justification

A MTK_JUSTIFY_LEFT MTK_JUSTIFY_TOP

B MTK_JUSTIFY_CENTER MTK_JUSTIFY_TOP

C MTK_JUSTIFY_RIGHT MTK_JUSTIFY_TOP

D MTK_JUSTIFY_LEFT MTK_JUSTIFY_CENTER

E MTK_JUSTIFY_CENTER MTK_JUSTIFY_CENTER

F MTK_JUSTIFY_RIGHT MTK_JUSTIFY_CENTER

G MTK_JUSTIFY_LEFT MTK_JUSTIFY_BOTTOM

H MTK_JUSTIFY_CENTER MTK_JUSTIFY_BOTTOM

I MTK_JUSTIFY_RIGHT MTK_JUSTIFY_BOTTOM

Table 1. Overlay Justification


Application Note

12

(0,0)

(0,M) (N,M)

(N,0)

A

D

G

B

E

H

C

F

I

Modifying the Overlays

An application may want to modify an overlay, such as an overlay’s position, size, or duration.

If a different image is required on a device, the application first removes an existing overlay by calling the sm_RemoveOverlays() function and then applies a new overlay using the sm_AddOverlays()function.

If an overlay’s position or size is to be changed, a call to the sm_RemoveOverlays() function is not required. Overlay location changes are accomplished by resetting the bounding frame on the overlay snapshot and calling the sm_AddOverlays() function on the snapshot. This is discussed in following section.

Bounding Frame Related Modifications

Several modifications can be made to an overlay that has already been applied to a media device. Some modifications are performed by manipulating the bounding frame set on the overlay.

Figure 13 shows a modification of an image’s placement on a media stream. The initial image was overlaid in the bottom-right corner. The image is being moved from bottom-right to upper-left of the video frame. After applying the initial overlay using the sm_AddOverlays() function, the application received SMEV_ADD_OVERLAY event. The event contained hDeviceSnapshot. Now the application wants to change the position of the image.

Since the overlay snapshot is a separate overlay from the template used to add the overlay, it has a separate bounding frame than the original template. To modify the bounding frame associated with the overlay snapshot, the application developer would need to make a call to the ob_GetOverlayBoundingFrame() function. Then mtk_SetFramePosition() and mtk_SetFrameSize() functions can be used with that newly acquired handle. The application must then use hDeviceSnapshot (from SMEV_ADD_OVERLAY) to apply the bounding frame change to the specific media device’s media stream.

Note that since the snapshot’s bounding frame was modified, no other overlay’s bounding frames are affected. A call to the sm_AddOverlay() function must be called on those devices using their own hDeviceSnapshot overlay handles and those overlays’ bounding frames.

Note that in the example shown in Figure 14, an existing overlay need not be removed. Also, the overlay and bounding frame templates do not need to be rebound.

Other Functionalities

Animation

Concepts presented so far in this application note can be used to create rudimentary animation effects. Some of the building block concepts that can be used to create animation effects are:

Creating a series of Overlay Templates and Bounding Frame Templates as shown in Steps 1 and 2 of Figure 6•

Bounding frame modification: position•

Bounding frame modification: size•

Timers

Timers are the key to creating animation effects. Timers are not provided by Dialogic® APIs. A timer facility to drive MTK API for animation effects can be created for an application.


Application Note

13

Using Timers

Although a maximum of two overlays may be added at a time, an application can create a series of overlay templates and bounding frame templates. Then, the application may remove one or both overlays that are currently applied and add the next overlay(s) from the series of pre-created templates. Using application-level timers, the application can remove or modify overlays as discussed in previous sections.

Cleaning Handles

MTK API provides other features besides image overlay. Due to this flexibility, many handles are tracked by the MTK library. These handles should be closed by the application at an appropriate time.

Figure 14 shows when different handles can be closed. Some handles (hBoundingSnapshot and hDeviceSnapshot) are not required to be closed. The MTK library takes care of closing the snapshot handles, such as those created by calls to the ob_CreateImageOverlay() function, and in the event data associated with calls to the sm_AddOverlays() function.


Application Note

14

hImageTemplate,hMedia,

hOvlTemplate,hBoundingFrameTemplate

sm_AddOverlays() →SMEV_ADD_OVERLAY →hDeviceSnapshot1…hDeviceSnapshot2

mtk_SetFramePosition(hSnapshotBoundingFrame, …)

mtk_SetFrameSize(hSnapshotBoundingFrame, …)

sm_AddOverlays(SRL_DEVICE,{direction, hDeviceSnapshot})…

mtk_DestroyMediaTemplate(hImageTemplate),mtk_DestroyMediaTemplate(hMedia),

ob_DestroyOverlayTemplate(hOvlTemplate),mtk_DestroyFrameTemplate(hBoundingFrameTemplate),

hSnapshotBoundingFrame = ob_GetOverlayBoundingFrame(hDeviceSnapshot1)

sm_RemoveOverlays(hDevice,hDeviceSnapshot1…hDeviceSnapshot2)

No need for these steps.

hBoundingSnapshot = ob_SetOverlayBoundingFrame(hOvlTemplate,

boundingFrameTemplate)

Figure 13. Bounding Frame Modification

Figure 14. Handle Cleanup

Demo Application

A sample application is provided to demonstrate the image overlay functionality of MTK API. This application is called img_ovl. It sets up a chain of devices as shown in Figure 15. This chain of devices includes two MM and two IPM devices.

The MM and IPM devices are routed to each other using the dev_PortConnect() function with DMFL_TRANSCODE_ON flag. The two IPM devices stream to each other using four RTP streams. Each IPM transmits a video and audio RTP stream for a total of two streams per IP media device.

One of the MM devices (mmB1C1) used for playing a media file contains audio and video. This media stream is transmitted to peer IPM device (ipmB1C1). This IPM device in turn sends RTP streams to neighboring IPM device (ipmB1C2) using the TCP/IP stack. The receiving IPM device then transmits the media stream to the second MM device (mmB1C2). This second MM device records the incoming media stream to audio and video files.

The sample application applies the overlay on mmB1C1 using direction of eSM_OVERLAY_DIRECTION_DEVICE. This results in the overlay on top of media stream flowing from mmB1C1 to ipmB1C1. No other overlays are added by the demo application.


Application Note

15

...

hImageTemplate,hMedia,

hOvlTemplate,hBoundingFrameTemplate,

hBoundingSnapshot

sm_AddOverlays() →SMEV_ADD_OVERLAY →hDeviceSnapshot1…hDeviceSnapshot2

mtk_DestroyMediaTemplate(hImageTemplate),mtk_DestroyMediaTemplate(hMedia),

ob_DestroyOverlayTemplate(hOvlTemplate),mtk_DestroyFrameTemplate(hBoundingFrameTemplate),

hBoundingSnapshot(No action required)

sm_RemoveOverlays(hDevice,hDeviceSnapshot1…hDeviceSnapshot2)

...

Building the Application

The img_ovl application can be built with or without the support of the ImageMagick library. To build img_ovl with the ImageMagick library, see the Using Dialogic® Media Toolkit API and ImageMagick for Image Overlay on Video Streams application note (see the For More Information section). Briefly, ImageMagick allows an application to convert text string to an image file. If ImageMagick support is enabled, img_ovl can convert UTF-8 encoded files or ASCII to images and use those images as overlays. The demo is provided with Hindi and Chinese strings in UTF-8 encoded files. Those files are in srcMedia directory.

To build the application without ImageMagick support, type:

make clean

make

To build the application with ImageMagick support, first follow instructions in the Using Dialogic® Media Toolkit API and ImageMagick for Image Overlay on Video Streams application note to install various packages. Then type:

make clean

USE _ IMAGE _ MAGICK=1 make

Starting the Application

The overlay that is applied depends on the command line option. The application takes “-o” command line option. If option 1 is selected, srcMedia/hello.jpg file is used for the overlay. If option 2 or 3 is selected, UTF-8 encoded file in the srcMedia directory is converted into an image using ImageMagick and that image is applied as overlay. If option 4 is selected, MM channel’s name is converted into an image using ImageMagick and applied.

Application built without ImageMagick support:

o - [0, 1] == Overlay option

0 == No overlay

1 == hardcoded overlay, see source code


Application Note

16

MM device(mmB1C1)

IP media device

(ipmB1C1)

Ao

Vo

Ai

Ai Ao

Vi

Vi Vo

RAo

RAi

RVo

RVi

Device Connections:dev_PortConnect(DMFL_TRANSCODE_ON)

File I/O:mm_Play

RTP Streams

IP

MM device(mmB1C2)

IP media device

(ipmB1C2)

Ao

Vo

Ai

AoAi

Vi

VoVi

RAo

RAi

RVo

RVi

Device Connections:dev_PortConnect(DMFL_TRANSCODE_ON)

File I/O:mm_Record

Figure 15. Demo Application Device Setup

Application built with ImageMagick support:

o - [0, 1, 2, or 3] == Overlay option

0 == No overlay

1 == hardcoded overlay, see source code

2 == Hindi overlay

3 == Chinese overlay

4 == Channel name as overlay

The demo application also takes “-cN” option, where N represents number of MM+IPM pairs to open. By default, two pairs are opened, as shown in Figure 11.

Application Runtime

The img_ovl application takes keyboard entries for several tasks. When the application starts, it opens the devices, initializes the image overlays, and connects the devices using the dev_PortConnect() function. Once the initialization is done, keyboard entry is allowed. Valid keypress options are:

s – Start mm_Play(), ipm_StartMedia(), mm_Record(), apply overlays

t – Stop activity started by “s”. Overlays applied to devices are also removed

r – After “s” and before “t”, remove an overlay that has already been applied.

o – After “s” and before “t”, add an overlay

q – Quit the application

A user may want to experiment by changing the code such that overlay is applied to video streams received by mmB1C2 or by applying the overlay on top video streams entering or exiting the IPM devices.

Determining Image Attributes

Many commercial and open source tools are available to help

edit images as well as identify the image attributes. One tool

is “identify” and it is included with ImageMagick (see the

application note listed in the For More Information section).

Sample output of “identify”:

[root@localhost export]# identify

-verbose hello.jpg

Image: hello.jpg

Format: JPEG (Joint Photographic

Experts Group JFIF format)

<snip>

Geometry: 100x100+0+0

<snip>

Compression: JPEG

<snip>

[root@localhost export]#

Summary

This application note presented the steps for enabling the

creation of overlay and border templates, applying overlays

and modifying overlays, and binding overlay templates and

border templates. API call sequences were presented, along

with a demo application. Image overlays can be applied into

or out of media streams of Dialogic® media devices such as

MM, IPM, and M3G. The reader can opt to implement various

application use cases listed in the Introduction.


Application Note

17

Appendix A: Dialogic® Media Devices

Appendix A provides background on the Dialogic media devices.

A variety of media devices stream audio and video:

• IPM devices — IPM devices source and receive RTP streams. Because there can be many instances of IPM devices in a system, IPM devices are enumerated using ipmBxCy naming scheme, where x represents a virtual board and y represents a channel with a unique RTP network address. IPM devices are network devices.

• MM devices — Multimedia devices provide file and buffer input/output to other devices. MM devices are a resource for the network devices. Multiple MM devices can be connected to each other as well.

• M3G devices — M3G devices are capable of controlling 3g324m connections. M3G devices rely on other transport channels such as a dti channel (DS0) for 3G Release 99 networks or ipm channel (NbUP over RTP) for 3G Release 4.

A common feature of these devices is that they have audio and video ports. These ports are used to stream media between these devices. The devices can also stream from/to the external network. Some device types can stream directly to the external network, whereas other device types are treated as resources that are used by the network devices for streaming media.

Interdevice Connections

Devices with common port types can be connected to each other via the Device Management API. An example is shown in Figure 16. MM and IPM devices have audio and video input and output ports; these are labeled Ai, Ao, Vi, and Vo ports. Ai and Ao are the device’s audio input and output ports respectively. Similarly, the Vi and Vo ports are the video input and output ports of the device. The input port of a device is connected to output port of peer device.

Figure 16. SIP Video Portal

Another example of device connections is shown in Figure 17. In this figure, a 3G caller is connected to a video portal service. The video portal service is enabled by connecting a dti device with m3g device. This dti ← → m3g connection provides the transport of multiplexed 3g-324m data between the 3G caller and the m3g device. The m3g device then de-multiplexes the 3g-324m data into audio, video, and control channels. The audio and video data is then connected to an MM device for multimedia playback and/or recording.


Application Note

18

MM device(mmBxCy)

IP media device

(ipmBxCy)

A

Ao

V

Vo

Ai

Ao

Vi

Vo

RAo

RAi

RVo

RVi

Device ConnectionsFile I/O RTP Streams

IP

In Figure 17, the external network is Release 99-based, which uses TDM for bearers. The DS0 is connected using the CTBus ports of dti and m3g devices. If the external network was IP-based Release 4/5, the 3g-324m stream would be carried over RTP. In that case, an IPM channel would be used for 3g-324m transport. See Figure 18 for the port connections between various devices.

Figure 17. 3G Video Portal, Release 99

Figure 18. 3G Video, Release 4/5


Application Note

19

Mux3G device

(m3gBnTm)

TDM device

(dtiBxTy)

MM device(mmBxCy)

Ai

Vi

Ao

AiAo

Vo

ViVo

324mi

324mo

CTbus

CTbusRJ45

PSTN

File I/O

Mux3G device

(m3gBnTm)

IP media device

(ipmBxCy)

MM device(mmBxCy)

Ai

Vi

Ao

AAo

Vo

VVo

324m

324mo

CTbus

324m

324mo

NbUPo

NbUPiIP

File I/O

For More Information

A Zip file containing the sample applications can be downloaded at — http://www.dialogic.com/goto/?11034

Related Application Note

Using Dialogic® Media Toolkit API and ImageMagick for Image Overlay on Video Streams — http://www.dialogic.com/goto/?11011

Reference Material

Dialogic® Media Toolkit API: Library Reference — http://www.dialogic.com/manuals/docs/media_toolkit_api_v2.pdf


Application Note

20

www.dialogic.com

Dialogic Corporation

9800 Cavendish Blvd., 5th floor

Montreal, Quebec

CANADA H4M 2V9

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH PRODUCTS OF DIALOGIC CORPORATION OR ITS SUBSIDIARIES (“DIALOGIC”). NO LICENSE, EXPRESS

OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN A SIGNED AGREEMENT

BETWEEN YOU AND DIALOGIC, DIALOGIC ASSUMES NO LIABILITY WHATSOEVER, AND DIALOGIC DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/

OR USE OF DIALOGIC® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY

INTELLECTUAL PROPERTY RIGHT OF A THIRD PARTY.

Dialogic products are not intended for use in medical, life saving, life sustaining, critical control or safety systems, or in nuclear facility applications.

Dialogic may make changes to specifications, product descriptions, and plans at any time, without notice.

Dialogic is a registered trademark of Dialogic Corporation. Dialogic’s trademarks may be used publicly only with permission from Dialogic. Such permission may only be

granted by Dialogic’s legal department at 9800 Cavendish Blvd., 5th Floor, Montreal, Quebec, Canada H4M 2V9. Any authorized use of Dialogic’s trademarks will be subject

to full respect of the trademark guidelines published by Dialogic from time to time and any use of Dialogic’s trademarks requires proper acknowledgement.

The names of actual companies and products mentioned herein are the trademarks of their respective owners. Dialogic encourages all users of its products to procure all

necessary intellectual property licenses required to implement their concepts or applications, which licenses may vary from country to country.

This document discusses one or more open source products, systems and/or releases. Dialogic is not responsible for your decision to use open source in connection with

Dialogic products (including without limitation those referred to herein), nor is Dialogic responsible for any present or future effects such usage might have, including

without limitation effects on your products, your business, or your intellectual property rights.

Any use case(s) shown and/or described herein represent one or more examples of the various ways, scenarios or environments in which Dialogic products can be used.

Such use case(s) are non-limiting and do not represent recommendations of Dialogic as to whether or how to use Dialogic products.

Copyright © 2008 Dialogic Corporation All rights reserved. 08/08 11033-01

Small Logo

Medium Logo

Large Logo

Date post:	16-May-2015
Category:	Documents
Upload:	ronny72
View:	509 times
Download:	0 times

Using Dialogic® Media Toolkit API for Image Overlay on Video ...

Documents