Aug. 5, 2014

GPUImage IntroductionBruce Tsai

Core Image

✤ Included in iOS from 2011!

✤ 100+ built-in filters!

✤ Multithread by CPU

Drawbacks

✤ Customization limited!

- Subclass CIFilter in iOS!

- Create custom filters (shader) only in OSX!

✤ Performance not tuned!

- Address tuning tips in WWDC 2012 (session 511)3

GPUImage

✤ Open source (BSD) iOS framework!

✤ GPU-based image and video processing!

✤ From GPU-accelerated video processing project (2010)!

✤ https://github.com/BradLarson/GPUImage

4

Advantages

✤ Performance!

- High FPS in real-time preview!

✤ Customization!

- OpenGL ES shader code

5

Buffer Process Flow

VideoCamera Sepia

View

MovieWriter

BlendPicture

Source Filter Output

Code Snippet

GPUImageVideoCamera *videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPreset640x480 cameraPosition:AVCaptureDevicePositionBack];

GPUImageFilter *customFilter = [[GPUImageFilter alloc] initWithFragmentShaderFromFile:@“CustomShader”];

GPUImageView *filteredVideoView = [[GPUImageView alloc] initWithFrame:CGRectMake(0.0, 0.0, viewWidth, viewHeight)];

[videoCamera addTarget:customFilter];

[customFilter addTarget:filteredVideoView];

[videoCamera startCameraCapture];

Class Diagram

Procedures

1. Add effect to input buffer !

2. Save result to output buffer!

3. Set targets’ input buffer attributes!

4. Set targets’ input buffer!

5. Invoke targets’ operation9

Frame Buffer Object (FBO)

glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);

glBindTexture(GL_TEXTURE_2D, renderTexture);

glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, renderTexture, 0);

Buffer Binding in FBO

Frame BufferTexturePixel Buffer

Render destinationActual render target

Main memory buffer

Conversion to and from Texture

CVPixelBufferRef pxBufferRef = CMSampleBufferGetImageBuffer(imageBuffer);

CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, videoTextureCache, pxBufferRef, NULL, GL_TEXTURE_2D, GL_RGBA, width, height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &videoTextureRef);

...

// renderTargetRef is corresponding to renderTexture of FBO

[writerAdaptor appendPixelBuffer:renderTargetRef withPresentationTime:pts];

Threading Model

✤ Grand Central Dispatch (GCD)!

- Dispatch queue!

✤ runSynchronousOnVideoProcessingQueue!

✤ runAsynchronousOnVideoProcessingQueue!

✤ runSynchronousOnContextQueue!

✤ runAsynchronousOnContextQueue14

Demo

Reference

✤ https://github.com/BradLarson/GPUImage!

✤ http://blog.db-in.com/all-about-opengl-es-2-x-part-2/!

✤ https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40008793!

✤ https://developer.apple.com/library/ios/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_intro/ci_intro.html#//apple_ref/doc/uid/TP30001185