Image Scaling

Jackie Van Ryzin

http://compsci.snc.edu/cs460/vanrjl

Outline

• Project Description– Definition & Requirements– Solutions (what I did)– Exceptions (what I didn’t get to)– Methodology (how I did it)

• Demonstration – Zooming fun!• Learning & Development Process

– Strategies (what works for me)– Knowledge (useful CS concepts)– Extensions (where to from here)– Advice (for those yet to come)

• Q & A

Project Description

• Develop algorithms for real-time scaling that maintains quality

• Study and implement smooth Bresenham nearest neighbor interpolation method

• Look at ways to improve these methods• Develop new algorithms or alter existing• Implement a user interface that would

allow someone to zoom in and out as they watch a streaming video

Overall Plan

• Read, research, understand existing algorithms

• Implement those algorithms using general arrays

• Create program to read and alter images• Combine image program and algorithms• Develop interface to test and compare

algorithms and variations of existing one• Alter programs & interface to scale real-time

streaming video in addition to still images

In the beginning…

• Thiadmer Riemersma was unhappy with:– Quality of fast image scaling algorithms– Slow processing for high quality scaling

routines • He developed smooth Bresenham

algorithm (Dr. Dobb’s Journal, May 2002)

• Used concept of Bresenham’s line algorithm to scale a horizontal line of pixels

Background in Bresenham

• Determine which points of an n-dimensional grid to plot to best approximate a straight line between two given points (x0,y0) and (x1,y1)

• For each x between x0 and x1, choose row y that most nearly matches fcn value to plot pixel (x,y)

To determine y

• General line formula is

• Know column x so solve for row y as

• Can pre-calculate constant

• Since slope between 0 & 1 by assumption, after rounding we either use same y or add 1 to it

)()(

)(0

01

010 xx

xx

yyyy

0001

01 )()(

)((int) yxx

xx

yy

)(

)(

01

01

xx

yy

Tracking Error• User error value to track difference between

current y value and exact y value for current x• As increment x++, increase error by slope• When error > 0.5, line is closer to next y value

rather than current/previous– y++ and error--

Bresenham for Images

• Pixels are picked up from discrete positions in source image and placed at discrete locations in the destination image

• Nearest neighbor sampling– Magnifying: pixels duplicated– Minifying: pixels dropped– Both leave artifacts in destination image

• More accurate sampling requires reading pixels from fractional positions in source image

• Techniques exist that incorporate weighted averages – computation-intensive

A “Lightweight” Alternative

• Compromise between linear interpolation & coarse Bresenham scaling

• Destination pixel set to:– Value of closest source

pixel– Unweighted average of

neighbors– Based on proximity to

source pixel

Original

150% 75%

Smooth Bresenham (cont’d)

• Merit:– Standard (unweighted) averages = simpler

calculations– Increased performance speed – important for

real-time applications– Dropped pixels & excessive “jaggies” of fast

nearest-neighbor techniques (such as coarse Bresenham) are absent or kept to a minimum

• Adjust this algorithm for 2D images– Nested for loops– Call horizontal line scaling algorithm– Average scaled lines with each other

How we use this… 

                                                                            

b (0,0)

g (0,0)

r (0,0)

b (1,0)

g (1,0)

r (1,0)

b (2,0)

g (2,0)

r (2,0)

:

:

b (0,1)

g (0,1)

r (0,1)

b (1,1)

g (1,1)

r (1,1)

:

:

b (width-1,height-1)

g (width-1,height-1)

r (width-1,height-1)

• Use CV’s split fcn to get 3 single-channel images from a 3-channel image

• Run algorithm on each before remerging with CV’s merge fcn

• CAREFUL with averaging!

• OpenCV Platform• Images in IplImage

format which has char* pointing to data

• Data stored linearly as bytes as shown

Zooming!

• At first could capture keys (used 1 & 2) with cvWaitKey for user to scale up or down

• Discovered cvTrackBar– Attaches nicely to

window– Auto updates the

variable (scale factor)

– Very user-friendly & visually appealing!

Analysis• First scale image using Bresenham, then

“undo” the scaling by using Bresenham on scaled image to get back to original size

• Compare with original image (calculate average difference between pixels of original and “un-scaled” images)

• Do same with cvResize functions & compare results

I Wish I Could/Would Have...

• Been better able to convert bytes to ints and back– Would have condensed processing time by calling

algorithm only once (on 3-channel image rather than separately on 3 1-channel images)

• Been able to implement more alterations to the algorithm for comparison and analysis

• Looked into MIP mapping as mentioned in DDJ– Quick scaling by factor of 2

• Used actual calculations to determine speed rather than simply visual

Methods

• Some mini-lectures/discussions with Dr. Pankratz & Dr. McVey in beginning to understand basic algorithm

• Test algorithms on arrays of ints first! (it was a while before I tried images)

• Tried using VB at first because Professor Blahnik provided code for reading from a webcam– Converting algorithm into VB (pointers)– DLL to leave it in C code– Picture box didn’t resize when image did

Methods (cont’d)

• OpenCV– Started with sample program from developers– Strip it down to minimum with John’s help– Add features and functions in baby steps

• Always had two versions– Still images using LoadImage– Webcam using Capture

• Wrote functions in small test program to run on arrays of ints before adding to interfaces

Let’s Scale!

• Real-time scrolling from webcam– HALLELUJAH!\

scroll_webcam\scroll_webcam.dsw

• Compare with CV resizing using still image– HALLELUJAH!\diff_cv\diff_cv.dsw

• How well does it “undo” the scaling?– HALLELUJAH!\rescale_cmp\rescale_cmp.dsw

• Does a different threshold matter?– HALLELUJAH!\

cmp_threshold\cmp_threshold.dsw

Where did this stuff come from?

• Dr. Dobb’s Journal• OpenCV manual• Dr. Pankratz & Dr. McVey• Forums and online tutorials for

OpenCV• Practice, experimenting, trial-and-

error• John, Ted, Ryan P

Important CS Concepts

• Data Structures– Arrays, pointers, structures, chars vs. unsigned

chars!– Dynamic memory allocation & de-allocation

• Machine Org. & Assembly Language– Fast calculations for dividing & masking

• Programming Languages– Designing abstractly without a language

• All classes– General learning, experimenting, researching

techniques

Where to From Here?

• Be able to convert to array of ints & back to reduce calls to scaling algorithm

• Test & compare speed using actual clock calculations

• Find a better mathematical/computational method for comparing how “nice” a picture looks scaled

• Alter algorithm in some way to improve quality and/or speed

• Investigate MIP mapping and using a combination of MIP & Bresenham

For Those Who Follow• Meet with DCP and other profs OFTEN and right

away• Ask questions of anyone and everyone

– Someone may have a unique solution you wouldn’t think of

– Could make you discover a solution with a simple comment about how chars in C aren’t fun

• When you get stuck, admit it and ask for help• Write your journal! It keeps you accountable and

encourages progress.• Paper can be useful! Draw images, diagrams, etc.

It helps to see it before you program it• Baby steps…starting small is easier, and every

little milestone is rewarding. It also makes testing & debugging easier.

Q & A

Thanks to Dr. Pankratz and Dr. McVey for the guidance and ideas!

Thank you to classmates for ideas, answers, and suggestions.

http://compsci.snc.edu/cs460/vanrjl