Distortion CorrectionECE 6276 Project Review

Team 5:Basit MemonFoti KacaniJason HaedtJin Joo LeePeter Karasev

2 ECE 6276 Final Project Team 5

7/14/2009

Outline

• Objective/Motivation

• Movement of Data

• C code

• Conversion to Catapult C

• Initial Results

• Current Issues

• Plans for Optimization

• Schedule

• Meeting with Mike Bradley

3 ECE 6276 Final Project Team 5

7/14/2009

Objective

Given a distorted image with known size and known lens distortion parameter, generate an undistorted image.

4 ECE 6276 Final Project Team 5

7/14/2009

Motivation – Why?

• The formation of undistorted images can be described by a series of matrix multiplications

• Distortion makes it very difficult to calibrate a camera to measure geometry (depth, size, orientation, etc)

• Many applications in image processing and computer vision like structure estimation, image mosaicing, and ultimately vision-based control.

5 ECE 6276 Final Project Team 5

7/14/2009

Motivation contd..

Application: Measure motion and geometryProblem: Known geometry in the scene is warped, relationship between 3D and 2D points is nonlinear. Solution: Undo the distortion, so x2D = A * X3D

6 ECE 6276 Final Project Team 5

7/14/2009

Movement of Data

Input

RAM on

Chip

Input Buffer

1 Pixel/Line

Random Read

LUT

Algorithm Output

7 ECE 6276 Final Project Team 5

7/14/2009

C Code Reference Design

const double powercoeffs[256][5] = { 0.0001676486936455572,0.9984675943966409,0.004271662030922045,-0.004648466894072161,0.001731029514912795,0.0001678607731736833,1.004564329741083,0.004318757090371574,-0.008746108062078511,0.001856610695386462,…

I/O scheme: Write to output in raster order, read input randomly (From buffer). New input put into buffer in raster order.

Coordinate in buffer does not have closed form expression- approximate with power series created in matlab, store as lookup table.

void undistort_ref( const unsigned char pixels_in[PIXELS],

unsigned char pixels_out[PIXELS], bool reset, unsigned short kappa_idx );

Top-level Entry point- block of pixels in and out, reset, index of distortion level

if( (iid >= 0) && (jjd >= 0) && (iid < HEIGHT) && (jjd < WIDTH) )pixels_out[ i_*WIDTH + j_ ] = pixels_in[ (iid*WIDTH)

+ jjd ];else

pixels_out[ i_*WIDTH + j_ ] = 0;

Conditional Write operation (ensure in-bounds)

// read next framen = fread(pix_in, 1, PIXELS, fin);// do the work for this blockundistort_ref( pix_in, pix_out, 0, kappa_idx);// write what we undistortedn = fwrite( pix_out, 1, PIXELS, fout );

Code verification- load binary input vectors, run, save, compare to matlab generated binary files

8 ECE 6276 Final Project Team 5

7/14/2009

Conversion to Catapult C

• Catapult C does not support dynamic memory allocation

- use static arrays for memory

- simplify C code structure to avoid pointers

• Any value that needs to be stored must be declared as static or could be synthesized out

• Top-level default instantiation assumes that input and output ports are wires, must explicitly declare memories

• Casting for printing is not automatically supported, must use .to_int() and .to_double() functions

• Had to simplify C code structure to not use as many pointers as concept is not as well defined in Catapult C

• Verify correctness in Visual Studio and Catapult environment

9 ECE 6276 Final Project Team 5

7/14/2009

Test Vector – Original Image

10 ECE 6276 Final Project Team 5

7/14/2009

Test Vector – Distorted Image

11 ECE 6276 Final Project Team 5

7/14/2009

Test Vectors – Recovered Image

12 ECE 6276 Final Project Team 5

7/14/2009

Initial Catapult C Synthesis (1/3)

Image Size(Pixels)

Optimization Parameters

Optimization (Design Goal)

Area Score Latency Cycles /Throughput

Cycles

Maximum Delay (ns)

Slack (ns)

8x8 No Optimization Area 14713 911/1250 11.41 -1.41

8x8 No sqrt Area 3523 1625/2154 9.27 0.73

8x8 No sqrt Latency 16125 591/5910 11.41 -1.41

32 x 32 No sqrt Area 4002.33 25697/31906 12.18 -2.18

64 x 64 No sqrt Area 4077.14 102593/127298 12.18 -2.18

256 x 256 No sqrt Area 3906.16 1639169/2098434 8.94 1.06

640 x 480 No sqrt Area 4392.45 7681921/9526242 9.49 0.51

13 ECE 6276 Final Project Team 5

7/14/2009

Initial Catapult C Synthesis (2/3)

14 ECE 6276 Final Project Team 5

7/14/2009

Initial Catapult C Synthesis (3/3)

15 ECE 6276 Final Project Team 5

7/14/2009

Current Problems

• Top issue is Area – we can’t support large image sizes based on current algorithm implementation.

• How do we achieve the ability to stream images in and then pass through our algorithm? Need to segment our algorithm and stream in portions of a larger image.

• Need to allocate large arrays to RAMs, particularly our LUT.

16 ECE 6276 Final Project Team 5

7/14/2009

Project Timeline

Tasks Owner Description Week 1 Week 2 Week 3Algorithm Research ALL Research different papers on barrel distortion

MATLAB Implementation PeterMock up distortion algorithm using MATLAB and use image

processing libraries for ease of use

MATLAB/C++ Testbench Jason Take MATLAB results and convert them into C++ structure to

be used in C++ testbenchC++ Code Porting Jin Joo Port MATLAB implementation of undistortion to C++

Presentation 1 ALL Initial project reportCatapult C Porting Foti,Memon Port to supported CATAPULT datatypes

Catapult C Synthesis Jason See results of CATAPULT algorithmPlans for Optimization Foti, Memon Look at ways to optimize CATAPULT code

Presentation 2 ALLCompleted Catapult Code, Synthesis Results, Steps for

OptimizationCatapult C Optimization Jin Joo Complete optimization suggestionsAlgorithmic Optimization Peter, Memon Ways to optimize the math in algorithm

Tool Optimization Foti, Jason Pipelining, Unrolling, Architecture considerationsCatapult C Synthesis of Optimization Jin Joo See results of CATAPULT algorithm

Demo ALL Show CATAPULT demo and final HDL Presentation 3 ALL Final report out

17 ECE 6276 Final Project Team 5

7/14/2009

Plans for Optimization• Reformulate problem slightly to avoid expensive sqrt()

operation (requires a new LUT)

• Ensure that LUT is mapping to ROM and buffer is on-chip RAM, not I/O pins

• Determine required bit-width for math operation result and use minimum

• Zero-pad arrays to be a power of two (avoids extra MUX elements)

• Avoid division- add lookup for radius values

• Experiment with size of input blocks. rgb2yuv example seemed to indicate large blocks create low latency, high area designs.

• Choose pipelining interval, loop to unroll. Unroll low level, pipeline top?

18 ECE 6276 Final Project Team 5

7/14/2009

Meeting with Mike Bradley

• He showed us how to ensure that LUT is mapping to ROM in Catapult C GUI

• Determined which for loops to unroll and which to pipeline

• Discussed different types of technologies ASIC vs FPGAs

19 ECE 6276 Final Project Team 5

7/14/2009

Updated Catapult Synthesis

-Catapult C no longer crashes for large input images

-Significant improvement in area, latency, and throughput

-Processes 1 pixel per clock cycle

20 ECE 6276 Final Project Team 5

7/14/2009

Questions?

?