Post on 14-Dec-2015
transcript
GStreamer T-111.5350 Multimedia programming
Mika Ristimäki, Jenny Lahti
History
The project was founded in 1999 by Erik Walthinsen.
Later it was joined by the maintainer of GStreamer, Wim Taymans.
The first corporate sponsor was RidgeRun, that hired Erik Walthinsen to develop methods for embedding GStreamer in smaller devices
The GStreamer logo was designed by Brock A. Frazier, who worked for RidgeRun.
Introduction
GStreamer was created to serve as a host for multimedia applications.
It is written in the C programming language It is a cross-platform product, so it works
e.g. on Linux, Solaris, Mac OS X and Microsoft Windows.
It provides a framework of plugins, for data flow and media type handling.
It is created to handle both audio and video.
With GStreamer’s development framework you can write any type of streaming multimedia application.
License – LGPL
GStreamer is released under LGPL (GNU Lesser General Public License).
LGPL means that the GStremer software itself has copyleft rules which means that you have to keep the software free.
But you are allowed to create your own software with your own copyright rules and link it with GStremer.
For companies like Nokia this license gives a profitable basis for their own software. They can use GStremer as a multimedia framework, but own the copyright to their own software, because it is merely linked to GStreamer.
Applications
GStreamer’s most obvious use is creating a media player on top of it, because it supports many needed formats for this.
GStreamer can also be used to create more complex programs like video or audio editing applications.
We intend to use it to create an Internet radio.
N770, N800 and N810
Nokia’s open source product N770 Internet Tablet and its successors N800 and N810, are palm-sized tablet PC:s running Linux
The tablets use GStreamer as a multimedia framework This enables 3rd party developers to use on-
board DSP without extensive knowledge about its architecture
They support audio (MP3, MPEG4-AAC, WAV, AMR and MP2) and video (MPEG-1, MPEG-4, Real Video, H.263, AVI and 3GP).
They have a Real player and a Flash player
N770, N800 and N810
Lifetrak
A project at University of California, Department for Electrical Engineering by Sasank Reddy and Jeff Mascia.
Lifetrak uses Nokia 770 as a hardware platform for creating a portable music player.
Lifetrak enables a context-aware playlist that automatically selects music based upon the users location, the pace of his movement and the environment
Lifetrak has a simple learning mechanism that can adjust the selection of the music based on the user’s ratings.
Lifetrak
Lifetrak
Lifetrak’s playing functionality is built on top of GStreamer
Lifetrak supports song files in the MP3 format, and in the future also Ogg-Vorbis, WAV, AAC, and MP4 files. GStreamer can handle each of these compression standards
GStreamer is also used for setting the volume, getting the position, and setting the position of the current song.
Mad Hatter’s Cocktail Party A project at Standford University, Palo Alto
Research Center and Robotics Institute in Carnegie Mellon University
Project wants to enhance conference call type mobile communication
The idea is to make communication possible for several small groups in a single audio space and for these groups to be able to regroup when desired.
The communication model is similar to a cocktail party.
Mad Hatter’s Cocktail Party The idea is to analyse the audio, find these
groups and make each listener hear his own group with a higher sound level and dimm out the voices of the rest.
This simulates a situation where you stand closer to the ones you listen to.
This is done by analysing when people speak, assuming that people in the same group mainly speek in turns.
Mad Hatter’s Cocktail Party The server, that analyses the voices, is a
PC running Linux using GStreamer. The server receives audio streame from
each client, analyzes them, applyes audio effect and performs mixing, and finally transmits audio back to the listeners.
Usage
GStreamer is so called “pipelined” multimedia framework Pipe-line consists of sequence of operations
The basic building block of a pipeline is called an element Pipe-line consists of chain of these elements and data
flows through it. For example, a basic mp3 audio decoder
Element 1 Reads mp3 file from hard disk.
Element 2 Decodes the mp3 data to raw audio
Element 3 Sends the raw audio to soundcard
Usage
Every element may also have a different number of pads Sink (input) pad and source (output) pad. Pad is an element’s link to outside world Pads can also be dynamic so that a pad can be
randomly created and/or destroyed. Interconnected elements form a bin
Is also element Same operations that can be done to an
element can be done to a bin also Pipeline is a specialized type of bin so that the
top level bin has to be pipeline
Usage
A bus takes care of internal messaging in GStreamer.
A callback function can be defined that takes care of EOS or other errors messages and acts upon them.
Every pipe-line has a bus by default The developer should create a callback function for the
bus. When the main loop is running the bus is checked for
messages and when a new message is noticed the callback function will be called
Example
Example
Creating elements
Example
Creating elements
Setting filesrc element properties
Example
Creating elements
Setting filesrc element properties
Adding the callback function for bus
Example Adding the callback function for bus
Putting elements to a bin
Creating elements
Setting filesrc element properties
Example
Putting elements to a bin
Creating elements
Setting filesrc element properties
Linking elements together
Adding the callback function for bus
Example Callback funtion for error handling
Function for dynamic pad creation
References
[1] GStreamer Application Development Manual (0.10.12.1), Wim Taymans, Steve Baker, Andy Wingo, Ronald S. Bultje, Stefan Kost
[2] DSP video processing via open-source APIs, EETimes OnLine, Rishi Bhatteracharya, 30.10.2006 http://www.eetimes.com/news/design/showArticle.jhtml?articleID=193401461
[3] The Mad Hatter’s Coktail Party: A Social Mobile Audio Space Supporting Multiple Simultaneous Conversations, Paul M. Aoki, Matthew Romaine, Margaret H. Szymanski, James D. Thornton, Daniel Wilson and Allison Woodruff, Palo Alto Research Center, CCRMA, Dept. of Music Stanford University, Robotics Institute Carnegie Mellon University, Florida USA 5.-10.43.2003, Paper: Design for the Socially Mobile
[4] Lifetrak: Music In Tune With Your Life, Sasank Reddy, Jeff Mascia, NESL, Department of Electrical Engineering University of California, Los Angeles
[5] Multimedia Abstraction Architecture of Cross-platform Applications Based on IMS, Jie Chen, Master Thesis, Helsinki University of Technology / Department of Computer Science and Engineering, 2006
[6] Linux for suits: a first look at the Nokia 770, Doc Searls, 02/2006, Linux Journal, Volume 2006 Issue 142, Publisher: Specialized Systems Consultants, Inc.
[7] Wikipedia, The Free Encyclopedia, (GStreamer, LGPL), http://en.wikipedia.org/wiki/Gstreamer, http://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License