IQA 暑期班 2017 -...

Fei [email protected]

http://camalab.hdu.edu.cn

Hangzhou Dianzi University

Image Quality Assessment via Deep Learning 2

Contents

• Background

• Image Quality Assessment (IQA)

• Photo Quality Assessment (PQA)

• Biometric Quality Assessment (BQA)

• Discussions


Contents

• Background




• Discussions


Background

• Dramatically increasing amount of imagesn Devices: computers, mobile phones, cameras, monitors, …

n Applications: medias, websites, IM clients, …


Background

• Extensively existed distortionsn Processing: acquisition, compression, transmission,

reconstruction, …

n Distortions: blurring, JPEG compression, Gaussian noise, …


What is Quality?

• Various aspectsn Athletics: perceived beauty or image appeal

n Fidelity: deviation from the undistorted version

n Intelligibility: discriminability of image content

[Keelan,HandbookofImageQuality,2002][JanssenandBlommaert,JIST,1997]

[Janssen,Proc.IEEE,2001][Halonen,etal.,Proc.SPIE,2011]

Athletics FidelityUndistortedDistorted

Intelligibility


Background

• Demands for IQA metrics

Evaluation, control, and improve the perceptual quality of multimedia content, quality of service (QoS), image processing systems, …

[Z. Wang, IEEE SPM’11]

Diagram of IQA-based feedback optimization

method.

ImageProcessingSystem

UpdatingAlgorithm

IQAEvaluation

Input Output


Background

• Common Subjective IQA Method

1005 Excellent

4 Good

3 Fair

2 Poor

1 Bad

80

60

20

0

40

Quality Scale/Score

Categorical Continuous

Test image k Test image k+1

T1 = 10s Reference image

T2 = 3s Mid-gray image

T3 = 10s Test image

T4 = 10s Mid-gray image

[ITUR BT.500-13., 2012]


Contents

• Background




• Discussions


Image Quality Assessment (IQA)

• Traditional Objective Methodsn Mean Squared Error (MSE)

n Peak Signal-to-Noise Ratio

n Advantages

• Mathematically conventional

• Effective for white noise

n Disadvantage

• Lack of consideration of human visual property

∑∑= =

−=m

i

n

jjiRjiI

mnMSE

1 1

2),(),(1

⎟⎟⎠

⎞⎜⎜⎝

⎛⋅=

MSEIPSNR2max

10log10

[Wang&Bovik,IEEESPM09]

Brightened(MSE=309) JPEG(MSE=309)

Gaussiannoise(MSE=309) Gaussianblur(MSE=309)



• How to Construct Objective IQA Metrics?

KnowledgeaboutTheHVS

PerceptualModeling

VisualPhysology andPhychophysics

Knowledgemapexpressing theelementsrequiredtoconstructsuccessful IQAmodels.

[Wang&Bovik,SPM’11]

KnowledgeaboutImageSource

Deterministic Natural Scenestatistics (NSS)

KnowledgeaboutImageDistortions

ArtifactDetection

DistortionModelling

Model-basedImageQualityAssessment

DualProblems



• Classification of Objective IQA

Reduced-reference(RR)IQAmethods

No-reference(NR)IQAmethods

Full-reference(FR)IQAmethods

RR-IQAmethods

NR-IQAmethods

FR-IQAmethods

Referenceimage

Distortedimage

SubjectiveQualityScores

SupplementaryInformation

Distortedimage


DistortedimageReferenceimage




• FR-IQA: Structural Similarity Index (SSIM)

Referenceimage!

Distortedimage"

LuminanceMeasure:Meanintensity

#$ =1'( )*

+

*,-

ContrastMeasure:Standarddeviationofintensity

.$ =1

'−1( )* −#$ 0+

*,-

-0

StructureMeasure:Normalizedimage

)−#$ /.$

2 !, " = 2#5#6+ 8-#50 +#60 + 8-

9 !, " = 2.5.6 +80.50 +.60 +80

: !, " = 2.56 +8;.5.6 +8;

<<)= !," =2 !," > ? 9 !," @ ? : !," A

[Z.Wang,etal. TIP’04]

n Advantages: efficient and effective

n Extensions: CBM [X. Gao et al., LNAI’05], IW-SSIM[W. Zhou and Q. Li, TIP’11], RR-SSIM [A. Rehman

and Z. Wang, TIP’12], etc.



• FR-IQA: Deep Similarity (DeepSim) [F.Gao,etal. NEUCOM’17]



• RR-IQAn Application: wireless communication scenario

n Framework of RR-IQA System

Distortion Channel

Feature Extraction

Ancillary Channel

Feature Extraction

RR Quality Analysis

Sender Side Receiver Side

original image

quality meature

RR features

distorted image



• NR-IQAn Convolutional Neural Networks for BIQA (Kang et al. CVPR’14)

50 feature mapsInput patch

32

32 77

minpooling

maxpooling

convolutional 26

26

50

50800

ReLUs ReLUs

linear regressio

noutput(score)

800

a) The input is locally normalized 32�32 image patches;

b) The convolutional layer produces 50 feature maps each of size 26�26, followed by a pooling operation that reduces each feature map to one max and one min;

c) The last layer is a simple linear regression with a one dimensional output that gives the score.



• Performance Evaluationn Color Image Database TID2013

• 25 reference color images (512*368 pixels);

• 120 distorted versions of each reference image;

• 24 types of distortions, 125 images of each type;

• Totally 3000 images;

• MOS of each image is available.

[N. Ponomarenko, et al. EUVIP’13]



• Performance Evaluationn Scatter plots



• Performance Evaluationn Pearson linear correlation coefficient (PLCC),

n Spearman’s rank- order correlation coefficient (SRCC), and

n Kendall rank-order correlation coefficient (KRCC)



• Applications: IQA Guided CNNs Training



• Applications: GAN



• Applications: Compression Artifact Removal

Galteri, Leonardo, et al. “Deep Generative Adversarial Compression Artifact Removal." arXiv preprint arXiv:1704.02518 (2017).


Contents

• Background

• Visual Quality Assessment (VQA)



• Discussions


Photo Quality Assessment (PQA)

• What is Photo Aesthetic Quality?

Beautiful / Professional / Good Ugly / Amateurish / Bad

[Marchesotti et al. 2015]



• Challengesn Aesthetic Principles

n Subjective PQA Mechanism

n Personalized

n Correlated with various image attributes (e.g. emotion, scene, semantic, style)



• Research Groups

Prof. Tang Xiaoou, SeanFellow IEEE

Multimedia Laboratory

Xinmei TianAssociate Professor

Xin Lu



• AVA database (250k images)n Ratings, Average ratings, Semantics, Styles



• Crowd Opinion [Marchesotti et al. 2015]









• Style

low depth-of-field, single salientobject and rule-of-thirds

[Marchesotti et al. 2015]

black and white


PQA via Deep Learning

• Methodology

Y. Deng, C.-L. Chen, and X. Tang, “Image aesthetic assessment: An experimental survey,” 2016.

PQA via CNNs

Single-column CNNs

Multi-column CNNs

Multi-task CNNs



• Single-column CNNsn CNNs trained on ImageNet

n Transferred CNNs

• Pretrain in the PQA task

• Fine-tuning in the binary classification task of photo aesthetic (good or bad)




• Multi-column CNNsn global features + local features

n global features + attribute features




• Multi-task CNNsn Correlated with various image attributes (e.g.

emotion, scene, semantic, style)

• Task 1: Aesthetic prediction

• Task 2: Semantic prediction

• Task n: …




• Multi-task Multi-column CNNs Y. Deng, C.-L. Chen, and X. Tang, “Image aesthetic assessment: An experimental survey,” 2016.



• A-lampn Adaptive Layout-Aware Multi-Patch



• A-lamp

Pipeline of attribute-graphs construction. (a) Salient objects (labeled by yellow bounding boxes) are first detected by a trained CNN, and regarded as local nodes. The dashed green bounding box denote the overall scene, which served as a global node. (b) Local and global attributes are extracted from these nodes to capture the object topology and the image layout. (c) Attribute-graphs are constructed and (d) concatenated into an aggregation layer.



• Performance




• Applications

n Enhancement

n Retargeting

n Retrieval

n Mobile APPs


Image Enhancement

• EnhanceGAN

• Deng, Yubin, Chen Change Loy, and Xiaoou Tang. "Aesthetic-Driven Image Enhancement by Adversarial Learning." arXiv preprint arXiv:1707.05251(2017).


Image Enhancement

• Creatism

• Fang, Hui, and Meng Zhang. "Creatism: A deep-learning photographer capable of creating professional work." arXiv preprint arXiv:1707.03491(2017).

A panorama (a) is cropped into (b), with saturation and HDR strength enhanced in (c), and with dramatic mask applied in (d).


Contents

• Background




• Discussions


Biometric Quality Assessment (BQA)

• Definitionn Quality of a biometric

sample is a measure of its efficiency in aiding recognition of an individual, ideally, irrespective of the recognition system in use.

Sample images of varying quality.

(a) Fingerprint, (b) iris (from WVU multimodal database), and (c) face (from SCface and CAS-PEAL face databases)



• Pipeline of a typical biometric system.



• Face



• Face: ID



• Four primary image features for BQA



• BQA via LightCNN



• Face recognition/identification/verification

Outline of the DeepFace architecture.





• Applications: Utilizing BQA for context switching


Contents

• Background




• Discussions

Date post:	26-Mar-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

IQA 暑期班 2017 -...

Documents