Kurento cpmx

transcript

Real-time Multimedia Stream Processing Developing rich multimedia applications with Kurento

The Stream oriented GE: Developing rich multimedia applications with Kurento

igracia@naevatec.com h0p://www.kurento.org

Open APIs for Open Minds

h"ps://join.me/kurentoCPMX

Mul8media infrastructures for the Future Internet

Media is here Media got

Media got there

Analyze Transform

Transport

Enrich Augment Adapt

Sensors Context

Events Media is here

Enrich, augment, adapt, analyze, transform, store: what’s the problem?

Complexity

Future Internet

Mul8media Infrastructure

Simple Development

Kurento: the equa8on

• Interoperable media exchange (mul8plaUorm/mul8protocol) • WebRTC, RTP, HTTP (video tag), etc.

• Process media (Computer vision, augmented reality, media indexing, etc.) • Media and metadata recording and recovery • Transform and adapt media (H.264, H.263, VP8, Ogg, and others) • Media rou8ng and mixing • Etc.

Mul8media infrastructure

• REST API •  JavaScript API •  Java API

•  LGPL 2.1

Is distributed through a flexible FOSS license

What’s Kurento

Kurento Media Server (KMS): the nucleus of Kurento

•  KMS is a middleware for media streams – Receives the stream – Process the stream –  Issues the stream

Send Receive Analyze Augment Enrich

Transform Transcode Record Process Replicate

Media Source

Media Sink

The Media API: The API for accessing KMS capabili8es

Send Receive Analyze Augment Enrich

Transform Transcode Record Process Replicate

Media Source

Media Sink

Java Media API

JavaScript Media API

REST API

Applica8ons define the processing of streams geeng through KMS

Media API: Media Elements and Media Pipelines

SRC% Sink%

§ Media Element • Provides a specific media

functionality › Send/receive media › Process media › Transform media

• Exchange media through ›  Sources ›  Sinks

§ Media pipeline • Chain of media elements

implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Media API: trivial example (Java) MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(“file:///myFile.webm”).build(); H2pGetEndpoint h"pEndpoint = mp.newH"pGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(h"pEndpoint); h"pEndpoint.getUrl(); //URL where the media is made available

Media Pipeline (inside KMS)

H"pGetEndpoint

Media from file or URI

HTTP media streaming Si

PlayerEndpoint

Is that enough? Think about the WWW development model

Process WWW request -‐  DDBB access -‐  Authen8ca8on -‐  XML processing -‐  Etc.

HTTP request: I want this resource

HTTP response: The resource

•  Intui8on behind the WWW model – Client asks what it wants

•  GET /this_resource HTTP/1.1 – Server side APIs execute the associated processing

You develop this using … •  Servlet •  JSP •  ASP •  PHP •  Ruby on Rails •  CGI •  Etc.

We need an equivalent model: the role of the Signaling Plane

•  We need “something” with the capability of nego8a8ng what, when and how media is exchanged

•  A protocol for “saying” –  I want “this media” …

•  Iden8fica8on of the media to exchange –  File in hard-‐drive, IP camera, user, etc.

–  in “this way” … •  Iden8fica8on of the processing of media

–  Augmented, analyzed, etc.

–  with “this format” … •  Quality

–  Codec, screen-‐size, frame-‐rate, etc.

–  at “this moment” •  Stream control

–  Play, stop, start, pause, etc.

Don’t get it? think about WWW development again …

Process WWW request -‐  DDBB access -‐  Authen8ca8on -‐  XML processing -‐  Etc.

HTTP request: I want this resource

HTTP response: The resource

Process media request -‐  Media API -‐  DDBB access -‐  Authen8ca8on -‐  XML processing -‐  Etc.

Signaling request: I want this media

Signaling response: The media is here

This is how you create WWW applica8ons

(Servlets, ASP, PHP, Rails, etc.)

This is how you create Kurento enabled applica8ons:

Mul8media RTC is just another feature of your

applica8on

You, as a programmer, create this

Abstrac8ng the signaling plane: The Content Handler

Media API

REST API (Open API protocol) The Content Handler Equivalent to a Servlet/ASP/PHP script Exposes APIs for specifying: -‐  when receiving “this request”… -‐  … execute “this logic” Developer can use the media API

Code building the media pipeline and execu8ng the applica8on logic the

developer wants

Kurento Applica8on Server: The container of Handlers

•  Is an extension of Java EE technologies. •  Compa8ble with all Java EE Servlet

containers •  Hold Handlers

–  Your specific applica8on code •  Receives signaling requests:

–  I want “this media” in this way … •  Dispatches request to the appropriate

handler –  @annota8ons based mapping

•  Generate an answer showing “where and how to find the media” –  URL where media is located –  Nego8ated SDP

Java EE compaJble container

HTTP Servlet

SIP Servlet

Web services

Kurento REST API

Specific handler implementa8ons

Media API DD.BB.

Kurento ApplicaJon Server (KAS)

Other java APIs.

Kurento Architecture: pueng it all together

Kurento Media Server (KMS)

Receive Video

Augmented Reality

Send Video

Computer Vision

Video Playing and Recording

Java EE compaJble container

HTTP Servlet

SIP Servlet

Web services

Kurento REST API

Specific handler implementa8ons

Signaling and WWW traffic

Media API DD.BB.

Kurento ApplicaJon Server (KAS)

Other java APIs.

Applica8on execu8on flow Client Code

Applica8on Server (KAS)

Media Server (KMS)

I want this media in this way … (JSON-‐RPC request)

Commands reques8ng the crea8on of a pipeline

What you want is here … (JSON-‐RPC answer)

Media negoJaJon

Media exchange phase

Specific applica8on logic at the server-‐side

(Content Handler)

Media pipeline creaJon

Media exchange between client and server

Content Handler: trivial example //Specify the type of service provided by this handler: Player, Recorder, WebRTC, RTP, etc. @H2pPlayerService(path = "/player”) //Mapping of handler specified in path public class MyPlayerHandler extends H"pPlayerHandler { @Override public void onContentRequest(H"pPlayerSession contentSession) {

//Thie client wants the media this handler provides //Create the pipeline for providing the media

} @Override public void onContentStarted(H"pPlayerSession contentSession) {

//Media started flowing, you can execute addi8onal ac8ons } @Override Public void onSessionTerminated(H"pPlayerSession contentSenssion){

//Media exchange termianted, you can collect your resources }

Let’s develop with Kurento •  What you need

–  A Kurento instance •  You can install your own Kurento instance •  You can launch a Kurento instance at the FI-‐LAB (FI-‐WARE project)

–  h"p://lab.fi-‐ware.org •  Geeng help

–  FI-‐WARE catalogue •  h"p://catalogue.fi-‐ware.org/enablers/stream-‐oriented-‐kurento

–  Kurento web site •  h"p://www.kurento.org

–  Kurento mailing list •  h"ps://groups.google.com/forum/#!forum/kurento

–  Twi"er •  @kurentoms

•  FI-‐LAB –  Working instance of FI-‐WARE enabling free experimenta8on with

technology –  h"p://lab.fi-‐ware.org

•  Crea8ng a Kurento instance from an image –  Use latest version of Kurento images.

•  Crea8ng a Kurento instance using recipes –  Use Ubuntu 13.10 clear image –  Use latest version of Kurento recipes

•  Accessing the Kurento demo at FI-‐LAB –  h"p://kurento.lab.fi-‐ware.org:8080/fi-‐lab-‐demo

Kurento at the FI-‐LAB

Kurento Hello World: Playing a file with an HTML5 client

Media Pipeline

H"pGetEndpoint

Media from file or URI

PlayerEndpoint

Media API

REST API (Open API protocol)

Handler code I want “this media”

Media is “at this URL”

Playing a file with an HTML5 client: Handler code

@H"pPlayerService(path = "/player”) public class MyPlayerHandler extends H"pPlayerHandler {

@Override public void onContentRequest(H"pPlayerSession contentSession) throws Excep8on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h2p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setA"ribute("player", playerEndpoint); H2pGetEndpoint h2pEndpoint = mp.newH2pGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(h2pEndpoint); contentSession.start(h"pEndpoint);

PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getA"ribute("player"); playerEndpoint.play();

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerHandler.java

Playing a file with an HTML5 client: client code <!DOCTYPE html>

var conn; func8on start() {

var op8ons = { remoteVideoTag : "remoteVideo"

}; conn = new kwsContentApi.KwsContentPlayer("./player", op8ons);

} func8on terminate() {

conn.terminate(); }

</script> </head> <body>

<bu"on onclick="start();">Start</bu"on> <bu"on onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/player.html

Playing a file with an HTML5 client: See the example working

h"ps://www.youtube.com/watch?v=n5BQlhYgGSo

Media Pipeline

Adding Computer Vision

H"pGetEndpoint Media from file or URI

PlayerEndpoint

JackVaderFilter

Media API

Handler code I want “this media”

Media is “at this URL”

Adding Computer Vision: Handler code @H"pPlayerService(path = "/playerWithFilter”) public class MyPlayerHandler extends H"pPlayerHandler {

@Override public void onContentRequest(H"pPlayerSession contentSession) throws Excep8on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h"p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setA"ribute("player", playerEndpoint); JackVaderFilter filter = mp.newJackVaderFilter().build(); H"pGetEndpoint h"pEndpoint = mp.newH"pGetEndpoint().terminateOnEOS().build(); filter.connect(h"pEndpoint); playerEndpoint.connect(filter); contentSession.start(h"pEndpoint);

PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getA"ribute("player"); playerEndpoint.play();

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java

Adding Computer Vision: Client code <!DOCTYPE html> <html>

}; conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", op8ons);

} func8on terminate() {

conn.terminate(); }

</script> </head> <body>

<bu"on onclick="start();">Start</bu"on> <bu"on onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/playerFilter.html

Adding Computer Vision: See the example working

h"ps://www.youtube.com/watch?v=yJAQs23eoXw

WebRTC loopback

Kurento Media Server

WebRTC Streaming

Media API

JSON-‐RPC API

Handler code I want “this media” (offer SDP)

Media is “here” (answer SDP)

Kurento ApplicaJon Server

WebRtcEndpoint

Client Browser

WebRTC loopback: Handler code

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcLoopback.java

@WebRtcContentService(path = "/webRtcLoopback") public class MyWebRtcLoopback extends WebRtcContentHandler {

@Override public void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); webRtcEndpoint.connect(webRtcEndpoint); contentSession.start(webRtcEndpoint);

WebRTC loopback: client code

<!DOCTYPE html> <html>

var op8ons = { localVideoTag : "localVideo", remoteVideoTag : "remoteVideo"

}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcLoopback", op8ons); }

func8on terminate() { conn.terminate();

} </script>

</head> <body>

<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcLoopback.html

WebRTC loopback: see the example working

h"ps://www.youtube.com/watch?v=HaVqO06uuNA

WebRTC recording

Media Pipeline

WebRTC Streaming

Media API

Handler code I want “this media (SDP)”

Media is “at here (SDP)”

Media to

file or URI

WebRTC recorder: Handler code

Source h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java Source of handler playing the recorded video: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerRecording.java

@WebRtcContentService(path = "/webRtcRecorder") public class MyWebRtcRecorder extends WebRtcContentHandler {

@Override public void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); RecorderEndpoint recorderEndpoint = mp.newRecorderEndpoint("file:///tmp/recording").build(); contentSession.setA"ribute("recorder", recorderEndpoint); webRtcEndpoint.connect(webRtcEndpoint); webRtcEndpoint.connect(recorderEndpoint); contentSession.start(webRtcEndpoint);

} @Override public void onContentStarted(WebRtcContentSession contentSession) {

RecorderEndpoint recorderEndpoint = (RecorderEndpoint) contentSession.getA"ribute("recorder"); recorderEndpoint.record();

WebRTC recording: client code

}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcRecorder", op8ons); }

} </script>

</head> <body>

<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <a href="./playerRecording">Play recorded content</a> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcRecorder.html

WebRTC recording: see the example working

h"ps://www.youtube.com/watch?v=ogN81PGkMuE

WebRTC one-‐to-‐many (Handler omi"ed for simplicity)

Media Pipeline

WebRTC one-‐to-‐many: Handler code

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcOneToMany.java

@WebRtcContentService(path = "/webRtcOneToMany") public class MyWebRtcOneToMany extends WebRtcContentHandler {

private WebRtcEndpoint firstWebRtcEndpoint; private String sessionId;

@Override public synchronized void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {

if (firstWebRtcEndpoint == null) { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); firstWebRtcEndpoint = mp.newWebRtcEndpoint().build(); sessionId = contentSession.getSessionId(); contentSession.releaseOnTerminate(firstWebRtcEndpoint); firstWebRtcEndpoint.connect(firstWebRtcEndpoint); //Loopback contentSession.start(firstWebRtcEndpoint);

} else { MediaPipeline mp = firstWebRtcEndpoint.getMediaPipeline(); WebRtcEndpoint newWebRtcEndpoint = mp.newWebRtcEndpoint().build(); contentSession.releaseOnTerminate(newWebRtcEndpoint); newWebRtcEndpoint.connect(firstWebRtcEndpoint); firstWebRtcEndpoint.connect(newWebRtcEndpoint); //Latest client gives media to the master endpoint contentSession.start(newWebRtcEndpoint);

WebRTC one-‐to-‐many: client code

}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcOneToMany", op8ons); }

} </script>

</head> <body>

<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcOneToMany.html

WebRTC one-‐to-‐many: see the example working

h"ps://www.youtube.com/watch?v=TBkrl3fmHWI

WebRTC game (Handler omi"ed for simplicity)

Media Pipeline

WebRTC game: Handler code

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcWithFilters.java

@WebRtcContentService(path = "/webRtcWithFilters") public class MyWebRtcWithFilters extends WebRtcContentHandler {

@Override public void onContentRequest(WebRtcContentSession contentSession)throws Excep8on {

final MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); final WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); final PointerDetectorFilter pointerDetectorFilter = mp.newPointerDetectorFilter().build(); final FaceOverlayFilter faceOverlayFilter = mp.newFaceOverlayFilter().build(); PointerDetectorWindowMediaParam start = new PointerDetectorWindowMediaParam("start", 100, 100, 280, 380); start.setImage("h"p://ci.kurento.com/imgs/start.png"); pointerDetectorFilter.addWindow(start); pointerDetectorFilter.addWindowInListener(new MediaEventListener<WindowInEvent>() { public void onEvent(WindowInEvent event) {

faceOverlayFilter.setOverlayedImage( "h"p://ci.kurento.com/imgs/mario-‐wings.png", -‐0.35F, -‐1.2F, 1.6F, 1.6F); }

}); webRtcEndpoint.connect(pointerDetectorFilter); pointerDetectorFilter.connect(faceOverlayFilter); faceOverlayFilter.connect(webRtcEndpoint); contentSession.start(webRtcEndpoint);

WebRTC game: client code

}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcWithFilters", op8ons); }

} </script>

</head> <body>

<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcFilters.html

WebRTC game: see the example working

h"ps://www.youtube.com/watch?v=5eJRnwKxgbY

Complex applica8on

Result: h"ps://www.youtube.com/watch?v=PYCw9-‐4oWWA

WebRtcEndpoint

MirrorFilter PointerDetectorFilter

ChromaFilter FaceOverlayFilter

Media Pipeline

•  Beyond video-‐conferencing: receiving events from the media stream –  Immersive and personalized adver8sing –  Games and entertainment

Events (bu"ons)

Game or adver8sement logic

Change face Change background

Collabora8ons welcome h"p://www.github.com/kurento

Thank you very much for your a"en8on

Sugges8ons, comments and complains: igracia@naevatec.com

Kurento cpmx

Technology