GROUP MAD COMPETITION

- A NEW METHODOLOGY TO COMPARE

OBJECTIVE IMAGE QUALITY MODELS

Kede Ma, Qingbo Wu, Zhou Wang, Zhengfang Duanmu,Hongwei Yong, Hongliang Li and Lei Zhang

June 28, 2016

Image Quality Assessment (IQA)

PurposeCreate objective models to predict human perception of image quality.

QuestionWith a significant number of IQA models available, how to fairly comparetheir performance?

2 / 13

Evaluating IQA Models

Conventional Evaluation MethodologyProve them by computing correlation metrics between subjective assessmentand objective model predictions.

ProblemEnormous image space

……

……

……

……

Subjective test

Unaffordable

3 / 13

MAximum Differentiation (MAD) Competition

Merits of MADDisprove IQA models by synthesizing strongest “counter-examples”.

Counter-examples search

�������������� �������������������� �������� �� ����������������� ��

�� �������������������� ��� �������������������� ��� �������������������� �

4 / 13

MAximum Differentiation (MAD) Competition

MAD Competition

Reference

image

B

C

D EA

Best SSIM for

fixed MSE

Worst SSIM for

fixed MSE

Best MSE for

fixed SSIM

Worst MSE for

fixed SSIM

Initial

distortion

5 / 13

Group MAD (gMAD) Competition

Attacking-Defending Game between Models

(a) An image collection

(b) Subset of images that

have the same PSNR

(c1) Best MS-SSIM image

(c2) Worst MS-SSIM image

(d1) Best BIQI image

(d2) Worst BIQI image

(e1) Best M3 image

(e2) Worst M3 image

image grouping by

PSNR (defender)

Pair sampling by other

models (attackers)

MS-SSIM

(attacher1)

BIQI

(attacher2)

M3

(attacher3)

6 / 13

Group MAD (gMAD) Competition

Subjective Testing

7 / 13

Group MAD (gMAD) Competition

Performance Measures1 Aggressiveness: How successful of a model at attacking another model?2 Resistance: How successful of a model at defending the attacks from

another model?

Global RankingThe Global rankings obtained by aggregating the aggressiveness matrix A andResistance matrix R.

8 / 13

Waterloo Exploration Database

4K+ source and ∼100K distorted images

Human Animal Plant Landscape

Cityscape Still-life Transportation

9 / 13

Applying gMAD to Waterloo Exploration Database

Pairwise Comparison between 16 Models

MS-SSIM

CORNIAFSIM

PSNRNIQE

ILNIQESSIM

TCLTLPSI

QAC

BRISQUENFERM

BIQI

BLIINDS-II M3

DIIVINE

MS-SSIM

CORNIA

FSIM

PSNR

NIQE

ILNIQE

SSIM

TCLT

LPSI

QAC

BRISQUE

NFERM

BIQI

BLIINDS-II

M3

DIIVINE

-60

-40

-20

0

20

40

60

Figure: Aggressiveness matrix

MS-SSIM

CORNIAFSIM

PSNRNIQE

ILNIQESSIM

TCLTLPSI

QAC

BRISQUENFERM

BIQI

BLIINDS-II M3

DIIVINE

MS-SSIM

CORNIA

FSIM

PSNR

NIQE

ILNIQE

SSIM

TCLT

LPSI

QAC

BRISQUE

NFERM

BIQI

BLIINDS-II

M3

DIIVINE -60

-40

-20

0

20

40

60

Figure: Resistance matrix

10 / 13

Applying gMAD to Waterloo Exploration Database

Global Ranking Result of 16 Models

Glo

bal r

anki

ng s

core

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

M3

DIIVINE

NFERM

BLIINDS-II

BRISQUEBIQI

QACTCLT

LPSISSIM

PSNRNIQE

ILNIQEFSIM

CORNIA

MS-SSIM

ResistanceAggressiveness

11 / 13

Applying gMAD to Waterloo Exploration Database

Observations1 FR-IQA models are more competitive;2 MS-SSIM and FSIM are top performing FR-IQA models;3 CORNIA and ILNIQE are top performing NR-IQA models;4 Machine learning based IQA models generally do not perform well.

12 / 13

Thank you

13 / 13