Image Deblocking Using Local Segmentation

Lukasz Kizewski

Supervisor: Dr. Peter Tischer

Second Examiner: Dr. Andrew Paplinski

Presentation Outline

Introduction Lossy Image Compression - JPEG Discrete Cosine Transform (DCT) Subbands

Present Research Research gap

DCT coefficient study Deblocking Filter Conclusions Further Research

Lossy Image Compression

JPEG, MPEG is lossy Maintaining high image quality and high compression

ratios is a major, widespread issue Used everywhere!

Internet, digital photography, digital camcorders, DVDs,Digital TV, video telephony/conference, mobile phones, etc.

JPEG/MPEG uses ‘Transform Coding’ technique coupled with data quantization JPEG/MPEG uses Discrete Cosine Transform (DCT) DCT is energy-preserving and reversible Quantization step is the lossy part

Discrete Cosine Transform

Image divided into 8x8 pixel blocks DCT applied to each block independently Block decomposed into basis functions

First basis function (0,0) termed ‘DC coefficient’ Remaining 63 basis functions termed ‘AC coefficients’

[1]

DC coefficient- Average brightness

AC coefficients-White: Add to average-Black: Subtract from average

DCT - Subbands

For every 8x8 pixel block, output of DCT is64 DCT coefficients Each coefficient corresponds to one basis function “Image” of 1 DCT coefficient termed a “subband”

[1]

2x2 blockDCT example

Subbands

First 4 (out of 64) subbands of ‘Lenna’

[1]

After performing the DCT, resulting coefficients are quantized Divided by quantum value, rounded to integer

Quantum value dictated by ‘quality’ parameter and quantization table

High-order DCT coefficients more severely quantized (usually to zero)

During de-quantization, mid-point of quantization interval is chosen Usually incorrect

DCT Coefficient Quantization

Q=10

5 14

1

5 1410

JPEG – Decompression Quality

JPEG decoded image at different ‘quality’ parameter settings:

Q = 100 Q = 50 Q = 1

JPEG Image artifacts

Incorrectly reconstructing DCT coefficients results in unwanted image artifacts: Smooth regions are blocky, edges are jagged,

discontinuities appear near edges Aim of project – decrease severity of artifacts

Smooth region(shoulder)

Staircase effect(edge of hat)

Ringing effect(edge of mirror)

Present Research

Dozens of filters exist to increase the quality of highly-compressed imagesSome filter DCT coefficients (subbands)Some filter reconstructed pixel valuesOthers filter bothMost filters target only one type of image

artifact Some filters reduce one type of artifact whilst

making another more prominent

Research Gap

Natural/photographic images have high correlation between neighbouring pixels Neighbouring pixels are similar in brightness Property fails when edge in image is encountered

Lack of image segmentation in most filters Results in blurred/smoothed edges

Encountering an edge implies more than one segment Segments should be filtered independently

Filling the Research Gap

This project differs in that:Local segmentation is used

Pixels possibly split into 2 groups

Each segment filtered independently“Do No Harm” policy used – avoid further

image quality degradation If filtering model fails, it’s filtered value is

disregarded Worst-case scenario – image is not filtered at all

Filling the Research Gap (cont.)

Filter can be applied toDCT Coefficients (filtering subbands)Pixels (filtering an image)Pixels may be filtered after subbands are

reconstructed as accurately as possible Quality loss occurs at subband level

First filtering step should be reconstructing DCT coefficients better

DCT Coefficient Study

Determine each subband’s contribution to overall image quality

Selected subbands were not quantized Simulates subbands being reconstructed perfectly Subband’s contribution measured by increase in

Peak Signal-to-Noise Ratio (PSNR) PSNR: Logarithmically-scaled, mean-squared-error metric NOTE: PSNR value doesn’t always reflect viewer-subjective

image quality assessment

DCT Coefficient Study (cont.) Results of study:

Image Deblocking Filter

Main goal is to reconstruct DCT coefficients better to reduce severity of image artifacts

DC subband filtered DCT study shows DC subband is best

filter candidate A DCT coefficient is a subband “pixel”

Uses 3x3 weighted mask Mask center is pixel being filtered Mask scans entire image, filtering

each pixel

1 2 1

2 4 2

1 2 1

3x3 filtering mask

DC subband of ‘Lenna’

“Do No Harm”

Filter employs “Do No Harm” (DNH) policy If a filtered value is implausible, reject it and

leave value unfiltered Implausible pixel value is one that falls

outside the quantization interval Quantization interval: midpoint +/- ½ Quantum

Guarantees image quality cannot degrade further

Sequential Filter

Try 1-segment filtering Replace pixel with mask average If DNH not triggered, accept

Otherwise try 2-segment filtering Use average-value thresholding to

classify pixels in mask Replace pixel with average of

segment to which it belongs to If DNH not triggered, accept

Otherwise trigger DNH and leave unfiltered value

100 100 100

100 15 15

100 10 5

Weighted average= 50.94

100 100 100

100 15 15

100 10 5

Blue class average=12.78

Sequential Filter - Results

Blocky effect reduced to 8x8 pixels Contrast between blocks minimised Filtered image can be used as input to another filter

DC Subband Expansion

Resolution of DC subband is increased 8-fold To match the resolution of the original image Inverse-DCT modified to use expanded DC subband

Linear-interpolation used to fill the gaps between original DC subband values

Applies a “gradient” to DC subband

128x128 pixel image 128x128 expanded DC subband

DC subband

DCT DC expansion

DC Subband Expansion - Results

Smooth regions completely void of blocky artifacts Expansion applied to filtered DC subband

What about the edges?

‘Staircase’ and ‘ringing’ effects still visible in all results

Sequential deblocking filter inappropriate for AC subband filtering AC subbands have very low

inter-pixel correlation Filtering AC subbands with

this filter introduces ringing DNH isn’t triggered because

AC quantization intervals are very wide

What about the edges? (cont.)

AC subbands describe edges, with respect to DC value Modifying AC coefficients adds/removes edges/textures Filtered image (far right) shows a light edge added next

to the actual edge – a ringing artifact! AC filtering – topic for further research

Original Reconstructed Filtered-reconstructed

Unsuccessful Additions

Always use 2-segment filtering Contrast between blocks increased ‘Chessboard’ effect

Threshold segmenting If class representatives are close, treat mask as 1

segment Finding correct threshold value impossible

3-segment filtering If 2-segment filtering triggered DNH,

3-segment filtering was attempted Little-to-no improvement in image quality Concluded that 2-segment model is sufficient for

95% of cases

Unsuccessful Additions (cont.)

Overlapping mask filtering All pixels’ filtered values in a mask were recorded Filtered value equal to average of all possible

recorded reconstructions (most pixels had up to9 possible filtered values)

Little-to-no improvement in image quality Applying sequential filter to reconstructed pixel

values “Quantization interval” parameter not known Guessing interval resulted in:

Blurry images – blurred edges Overly-sharpened images – “cardboard cut-out” effect with

flat colors

Conclusions

Filtering DC subband has been successful in improving overall image quality As predicted by the DCT coefficient study Due to high inter-pixel correlation

AC subbands must be filtered in some other way This filter produces ringing artifacts

Due to averaging Due to little inter-pixel correlation Due to extremely wide quantization intervals – allowing large

change to an AC coefficient’s value Little or no information left in subband

Perhaps try median-filtering, instead of averaging May introduce blurring of edges

Conclusions (cont.)

Knowledge of a subband’s quantization interval aids in segmentation Maximum quantization error is known DNH “switches” between 1- or 2-segment filtering

Sequential (adaptive) filtering model proved successful DNH policy ensured no further image quality loss Sometimes too strict

Some blocks “don’t fit” in with surrounding pixels Conforms to JPEG specifications

Critical for video sequence coding (MPEG) – reconstruction errors propagate to subsequent frames

Future Research

AC subbands must be filtered in a different manner altogether Perhaps use DC subband segmentation to drive AC

subband filtering Resulting image from filtered,

non-expanded DC subband filter can be used as a starting point for a secondary filter This filter should target edge artifacts

Once edges are also filtered, DC expansion could be applied as a third step

Try median-filtering, instead of averaging

For More Information…

Consult Thesis See Website

http://www.csse.monash.edu.au/~kizewski/

E-mail me kizewski@mail.csse.monash.edu.au

Read referenced material Read filter source code Ask a question

[1]: Rabbani, M. and Jones, P. W. (1991). Digital Image Compression Techniques, Vol. TT7 of SPIE Tutorial Texts.[2]: http://www.mat.univie.ac.at/~kriegl/Skripten/CG/node53.html

Thank-You

Questions?

…and now, for some eye-candy!