SETTINGVIDEOQUALITY&PERFORMANCETARGETSFORHDRANDWCGVIDEOSERVICES

SEANT.MCCARTHY

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 2

TABLEOFCONTENTSINTRODUCTION.............................................................................................3QuantifyingHDRWCGVideoQuality&Distortions.......................................................3

ThePerformanceofExistingHDRVideoQualityMetrics...............................................4

BalancingPerformanceandComplexity.........................................................................5

CHARACTERISTICSOFHDRWCGVIDEO........................................................6TestSequences&Preparation.......................................................................................6

RepresentingImagesinTermsofSpatialFrequency......................................................6

ExpectableStatisticsofComplexImages.......................................................................7

PROPOSEDHDRWCGVIDEODISTORTIONALGORITHM...............................8SpatialDetail...................................................................................................................8

EffectofHEVCCompressiononSpatialDetailCorrelation..........................................11

UsingSpatialDetailtoProbeBright&DarkFeaturesandTextures.............................14

SpatialDetailCorrelationforHDRWCGFeaturesandTextures..................................16

WeightedMean-SquaredError....................................................................................17

Squared-ErrorDensity..................................................................................................18

CONCLUSION...............................................................................................19ABBREVIATIONS...........................................................................................21RELATEDREADINGS.....................................................................................22REFERENCES................................................................................................23

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 3

INTRODUCTIONHighDynamicRange(HDR)andWideColorGamut(WCG)canhaveabigpositiveimpactonaviewerbycreatingamoreconvincingandcompellingsenseoflightthanhaseverbeforebeenpossibleintelevision.Arecentscientificstudy1withprofessional-qualityStandardDynamicRange(SDR)andHDRvideosfoundthatviewerspreferHDRoverSDRbyalargemargin.Moreover,thestudyalsoshowedthatthemarginofpreferenceforHDRincreasedwithincreasingpeakluminance.

Whathappensthoughtoaviewer’squalityofexperiencewhenpristinehighqualityHDRcontentiscompressedfordistribution?WhathappenswhenHDRWCGcontentisconvertedtoSDRcontenttosupportlegacydisplaysandconsumerset-topboxes?DodistortionsandcompressionartifactsbecomemorenoticeableinHDR?DoesprocessedHDRlosesomeofitssparkleandbecomelessdiscerniblefromordinarySDR?

Videoqualityiseasytorecognizebyeye,butputtinganumberonvideoqualityisoftenmoreproblematic.ForHDR&WCGtheproblemisevenharder.HDR&WCGaresoperceptuallypotentbecauseevenrelativelyinfrequentfeaturessuchasspecularreflectionsandsaturatedcolorscanengageaviewer’sattentionfully.Yet,well-knownvideo-qualityscoringmethods,suchaspeaksignal-to-noiseratio(PSNR)andtheStructuralSIMilaritymetric2(SSIM),couldleadtowrongconclusionswhenappliedtotheperceptualoutliersinHDRWCGvideo.Withoutgoodvideo-qualitymetrics,cableoperatorscannotmakeinformeddecisionswhensettingbitrateandvideo-qualityperformancetargets,norwhenchoosingtechnologypartnersforHDRWCGservices.

WeneedawayofquantifyingdistortionsintroducedduringHDRWCGvideoprocessingthattakesintoaccountthewideluminancerangeofHDRvideoaswellasthelocalizedhighlights,deepdarks,andsaturatedcolorsthatgiveHDRWCGitsspecialappeal3.

Thispaperintroduceseasy-to-calculatequantitativemethodstoprovidecableoperatorswithvideo-qualitydatathatcanbeusedtomakeoperational,technological,andproductdecisions.Specifically,itpresentsmethodstoreportthelevelofoveralldistortionsinprocessedvideoaswellasthespecificdistortionsassociatedwithperceptuallyimportantbright&darkHDRfeaturesandtextureswithrespecttobothlumaandchromacomponents.Thepaper’sobjectiveistoshowdataandanalysisthatillustrateshowquantifyingHDRWCGvideodistortioncanbemadeaccurate,actionable,andpractical,particularlywhenMSOsconsiderthevarioustrade-offsbetweenbandwidth,technologyoptions,andtheviewer’sexperience.

QuantifyingHDRWCGVideoQuality&DistortionsThebestwaytoquantifyvideoqualityandviewerpreferenceistoperformsubjectivetestingusingestablishedtechniquesandexistinginternationalstandardssuchasITU-R

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 4

BT.5004andITU-TP.9105;butsubjectivetestingistooslowtobepracticalinmostsituations.Instead,anumberofobjectivevideoqualityassessmenttechniquesandmetricshavebeendevelopedoverthedecades6.Objectivevideoqualityassessmentreliesoncomputeralgorithmsthatcanbeinsertedintoproductionanddistributionworkflowstoprovideactionableinformation.Somevideoqualityalgorithms,suchasPSNR,areverysimple,butdonotcorrelatewellwithsubjectivescores7,8.Othersareverysophisticatedandincludemodelsofthehumanvisualsystem.Suchmetricsdoabetterjobofpredictingsubjectiveresults,butcansufferfromcomputationalcomplexitythatlimitstheiruniversalusefulness9.Stillsomeothervideoqualitymetrics,suchasSSIMandmultiscaleMS-SSIM10,haveemergedthatstrikeagoodandusefulbalancebetweencomplexityandabilitytopredicthumanopinionswithreasonableaccuracy.

Anotherimportantclassofvideoqualitymetricsanalyzesprimarilythesignalcharacteristicsofimages,thoughtheyoftenalsoincludesomeaspectofthehumanvisualsystem.TheVIFmetricdevelopedbySheikhandBovik11,forexample,incorporatesthestatisticsofnaturalscenes12.NillandBouzas13developedanobjectivevideoqualitymetricbasedontheapproximateinvarianceofthepowerspectraimages.Lui&Laganiere14,15developedamethodofusingphasecongruencytomeasureimagesimilarityrelatedtoworkbyKovesi16,17andbasedontheproposalbyMorrone&Owens18andMorrone&Burr19andthatperceptuallysignificantfeaturessuchaslinesandedgesarethefeaturesinanimagewherethespatialfrequencycomponentscomeintophasewitheachother.Morerecently,Zhangetal.20leveragedtheconceptofphasecongruencytodevelopFSIM,afeaturesimilaritymetric.

Themetricweproposeinthispaperfallsinwiththeabovegroupofmetrics.Itsharesthesamemindspaceinthatitreferencesstatisticallyexpectablespatialfrequencystatisticsandthesignificanceofphaseinformationinanimage;butalsoitdiffersinseveralimportantaspects.Themetricweproposedoesnotrelyonphasecongruencybutratherona“SpatialDetail”signalthatcanbethoughtofasacombinationofthetruephaseinformationinanimageandthestatisticallyunpredictableinformationinanyparticularimage.The“SpatialDetail”signalcanbethoughtofasthecondensedessenceofanimagethathasthetwinadvantagesofbeingveryeasytocalculateandofprovidingaguidetothebrightanddarkfeaturesandtexturesthatgiveHDRWCGitsspecialappeal.

ThePerformanceofExistingHDRVideoQualityMetricsItwouldbesimpleifwecouldusetheSDRobjectivevideoqualitymetricswehavecometoknowsowelltoquantifyHDRvideoqualityalso.ItturnsoutthatobjectivevideoqualityassessmentforHDRisnotsimple.HDRvideoqualityassessmentneedseithernewalgorithmsandmetricsoranewmoreperceptuallymeaningfulwayofrepresentingimagedata.Perhapsbothwillbeneeded.

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 5

Hanhart,etal.1,recentlyreportedastudyofobjectivevideoqualitymetricsforHDRimages.Theylookedattheaccuracy,monotonicity,andconsistencyofalargenumberofbothlegacySDRandnewerHDR-specificmetrics21-24withrespecttoeachmetric’spredictionofsubjectivevideoqualityscores.TheyfoundthatmetricssuchasHDR-VDP-223andHDR-VQM24thatweredesignedspecificallyforHDRcontentwerebest.

Interestingly,Hanhartetal.alsofoundthattheperformanceofmostfull-referencemetrics,includingPSRNandSSIM,wasimprovedwhentheywereappliedtononlinearperceptuallytransformedluminancedata(PU25andPQ26)insteadoflinearluminancedata.AsimilarconclusionwasreportedearlierbyValenziseetal.27whousedaperceptuallyuniform“PUtransform”developedbyAydinetal.25toassesscompressedHDRimages.TheyfoundthatPU-basedPSNRandSSIMperformedaswellandsometimesbetterthanthemorecomputationallydemandingHDR-VDP21algorithm.AnotherstudybyManteletal.28alsoreportedthatperceptuallinearizationinfluencedtheperformanceofobjectivemetrics,thoughinthisstudyperceptuallinearizationdidnotalwaysimproveperformance.Rerabeketal..29extendedthestudyofobjectivemetricsbeyondstillimagestoHDRvideosequencesandfoundthatperceptuallyweightedvariantsofPSNR,SSIM,MSE,andVIFcorrelatedwellwithsubjectivescores,thoughHDR-VDP-2wasfoundtobethebestperformerstatistically.

BalancingPerformanceandComplexityObjectivevideoqualityalgorithmsshouldbeassimpleaspossibleandnosimpler.Complexmodelsofhumanvisionareimportantandhavetheirplace,butcanalsobecometoocumbersometobepracticallydeployedinproductionanddistributionofvideoprograms.Ontheotherhand,simplerfidelitymetricssuchasPSNR,SSIM,andMS-SSIMmightbesettingthebartoolowevenwithperceptuallylinearizedimagedata.

ThispaperproposesnewHDRWCGvideodistortionmetricsandanalgorithmthatisintendedtobesimple,fast,andprovideactionabledatatomonitorandimproveeverydayvideooperations.

ThevideodistortionassessmentmethodwepresentleveragesaframeworkofbiologicallyinspiredimageandvideoprocessingdevelopedbyMcCarthy&Owen30,31basedonstudiesofthevertebrateretinaandtheexpectablestatisticsofnaturalscenes.Thisbio-inspiredframeworkhasbeenleveragedpreviouslytodevelopaperceptualpre-processorusedinprofessionbroadcastencoders32tomakevideomorecompressiblewhileminimizingintroducedartifacts.Thedetailsofthetheoryarebeyondthescopeofthepaper,buttheapplicableelementsofthetheorycanperhapsbestbeexplainedbyconsideringvideointermsofspatialfrequency(seeFigure2).

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 6

CHARACTERISTICSOFHDRWCGVIDEOTestSequences&PreparationInthisstudy,weusedtheHDRWCGtestsequencesshowninFigure1.Thesesequenceswerecreatedbythe“HdM-HDR-2014Project”33,34toprovideprofessionalqualitycinematicwidegamutHDRvideofortheevaluationoftonemappingoperatorsandHDRdisplays.Allclipsare1920x1080p24andcolorgradedforRec.2020primaries&0.005-4000cd/m2luminance.TosimulatecableandpayTVscenarios,weconvertedtheoriginalcolorgradedframes(RGB48bitsperpixelTIFFfiles)toYCbCrv210format(4:2:210bit)usingtheequationsdefinedinITU-RBT.202035.AllvideoprocessingandanalysiswasperformedusingMatlab36,ffmpeg37,andx26538.

Figure1-HDRWCGTestSequencesUsedinthisStudy

RepresentingImagesinTermsofSpatialFrequencyAnimageisnormallythoughtofasa2-dimensionalarrayofpixelswitheachpixelbeingrepresentedbyred,green,andbluevalues(RGB)oralumaand2chromachannels(forexample,YUV,YCbCr,andmorerecentlyICTCP).Animagecanalsoberepresentedasa2-dimensionalarrayofspatial-frequencycomponentsasillustratedinFigure2.Thevisualpixel-basedimageandthespatial-frequencyrepresentationofthevisualimageareinterchangeablemathematically.Theyhaveidenticalinformation,justorganizeddifferently.

Figure2-RepresentationofaVideoFrameinTermsofSpatialFrequency

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 7

Spatial-frequencydatacanbeobtainedfromanimagepixelarraybyperforminga2-dimensionalFastFourierTransform(FFT2).Thepixelarraycanberecoveredbyperforminga2-dimensionalInverseFastFourierTransform(IFFT2).FFT2andIFFT2arewellknownsignalprocessingoperationsthatcanbecalculatedquicklyinmodernprocessors.

Inthespatialfrequencydomain,theinformationinanimageisrepresentedasa2-dimensionalarraycomplexnumbers;orequivalentlyasthecombinationofareal-valued2-dmagnitudespectrumandareal-valued2-dphasespectrum.(NotethatthelogofthemagnitudespectrumisshowninFigure2toaidvisualization.ThehorizontalandverticalfrequencyaxesareshownrelativetothecorrespondingNyquistfrequency(±1).)

Figure3-ThePhaseSpectrumTypicallyContainsMostoftheDetailsofanImage

Thephasespectrumcontainsmostofthespecificdetailsontheimage,asillustratedinFigure3.Onewaytothinkofthephasespectrumisthatitprovidesinformationonhowthevariousspatialfrequenciesinteracttocreatethefeaturesanddetailswerecognizeinimages18,19.Themagnitudespectrumtypicallycarrieslittleuniqueidentifyinginformationaboutanimage.Instead,itprovidesinformationonhowmuchoftheoverallvariationwithinthevisual(pixel-based)imagecanbeattributedtoaparticularspatialfrequency.

ExpectableStatisticsofComplexImagesImagesofnaturalsceneshaveaninterestingstatisticalproperty:Theyhavespatial-frequencymagnitudespectrathattendtofalloffwithincreasingspatialfrequencyinproportiontotheinverseofspatialfrequency12.Themagnitudespectraofindividualimagescanvarysignificantly;butasanensemble-averagestatisticalexpectation,itcanbesaidthat“themagnitudespectraofimagesofnaturalscenesfalloffasone-over-

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 8

spatial-frequency.”Thisstatementappliestobothhorizontalandverticalspatialfrequencies.

Figure4-Illustrationof“One-Over-Spatial-Frequency”MagnitudeSpectra

Figure4demonstratesthatindividualframesoftheHDRWCGtestsequencesusedinthisstudygenerallyadheretothe“one-over-spatial-frequency”statisticalexpectation.Theplotsalongthebottomofthefigureshowthevaluesofthemagnitudespectrumalongtheprincipalhorizontal(orange)andvertical(blue)axescorrespondingtothehorizontal(orange)andvertical(blue)arrowsinthemiddlerowofthefigure.

Itisworthnotingthattheexpectablestatisticsof“natural-scene”imagesarenotlimitedtopicturesofgrassandtreesandthelike.Anyvisuallycompleximageofa3-dimensionalenvironmenttendstohavetheone-over-frequencycharacteristic,thoughman-madeenvironmentstendtohavestrongerverticalandhorizontalbiasthanunalteredlandscape.Theone-over-frequencycharacteristiccanalsobethoughtofasasignatureofscale-invariance,whichreferstothewayinwhichsmallimagedetailsandlargeimagedetailsaredistributed.Imagesoftextandsimplegraphicsdonottendtohaveone-over-frequencymagnitudespectra.

PROPOSEDHDRWCGVIDEODISTORTIONALGORITHMSpatialDetailHDRisallaboutpreservingspatialdetail.Itisnotaboutbrighterpictures39,40,oratleastitshouldnotbe.ThewiderluminancerangeencodedbyHDRenablescrispspatialdetail

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 9

indarkregionsandbrighthighlightstoplayaroleinstorytellingthatisnotpossibleotherwise.Similarly,WCGisallaboutenablingcolorfulnessofspatialdetails.

Whatis“spatialdetail?”Weknowitwhenweseeit;butifwecan’tmeasureitquantitativelywecan’tmanageitsystematically.

Weproposethat“spatialdetail”canbequantifiedasthephaseinformationinanimagecombinedwiththestatisticallyunexpectablevariationsinthemagnitudespectruminformation.

Figure5-MethodofCalculatingtheSpatialDetailSignal

OurmethodofcreatingaSpatialDetailsignalisillustratedinFigure5.First,themagnitudeandphasespectraarecalculatedfromtheimagepixelarray(onlythelumachannelisshowninFigure5,butthemethodologymayalsobeappliedtothechromachannelor,alternatively,tothered,green,andbluechannels.)Next,apredeterminedarchetypeofthestatisticallyexpectableone-over-frequencymagnitudespectrumisdividedintotheactualmagnitudespectrumtoproduceastatisticallyweightedmagnitudespectrum.Third,thestatisticallyweightedmagnitudespectrumiscombinedwiththeactualphasespectrum.Finally,a2-dimensionalInverseFastFourierTransformisappliedtoproduceapixelarraythatwecalltheSpatialDetailsignal(seeFigure6).

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 10

Figure6-EnlargedViewoftheSpatialDetailSignalfortheLumaComponent

TheSpatialDetailsignalcanbethoughtofastheresultofa“whitening”process.However,atruewhiteningisasignalprocessingoperationthatresultsinexactlyequalmagnitudevaluesatallfrequencies.ThephaseimageshowninFigure3istheresultofatruewhiteningprocess.ItisperhapsmoreusefulandaccuratetothinkoftheSpatialDetailastheresultof“statisticallyexpectablewhitening”thatcontainstheresultofatruewhitening(thephaseimage)filteredbythestatisticallyunexpectablemodulationsofthemagnitudespectrum.Thedistinctionmightseemnuanced,yetthedifferencehaspracticalbenefits.Whereasthephaseimage(Figure3)isroughand“noisy”inawaythatobscurestherecognizabledetailsinanimage,theSpatialDetailsignal(Figure6)isasmoothlyvaryingmorerecognizabledualoftheoriginalimage.

TheSpatialDetailsignalmayalsobethoughtofastheresultofatrue2-dimensionaldifferentiationoftheimagepixelarray.TheSpatialDetailsignalisobtainedbydividingtheactualmagnitudespectrumbyaone-over-frequencyspectrum,whichisequivalenttomultiplyingtheactualmagnitudespectrumbyfrequency.Multiplicationbyfrequencyinthefrequencydomainisequivalenttodifferentiationinthepixeldomain.

ThedifferentiationcharacteristicoftheSpatialDetailisapparentinFigure7.Thelumavaluesoftheoriginalpixelarray(A)alongthemidline(dashedline)areplottedintheuppermiddlegraph(C).Thehistogramoftheallthelumavaluesoftheoriginalpixelarrayareplottedintheupperrightgraph(E).ThecorrespondingSpatialDetailsignal(B)valuesalongthemidlineareplottedinthelowermiddlegraph(D).ThehistogramofthealltheSpatialDetailvaluesareplottedinthelowerrightgraph(F).NotethattheSpatial

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 11

Detailvaluestendtoclusternearzeroanddeviatesignificantlyfromthezerolineonlywheretheoriginallumavalueschangesignificantly.

NotealsothattheSpatialDetailhistogramiscenteredonzeroandissymmetric,biphasic,andformsacompactpeakeddistribution.Conversely,theoriginallumavaluesarespreadout.ThesignificanceofthisdistinctionisthatthedistributionofSpatialDetailvaluesispreservedacrossimages.Thewidthofthehistogramchangesmoderatelyfromonevideosequencetoanotherbutretainsthestereotypicalcompact,peaked,biphasic,andsymmetricshape.Inotherwords,theSpatialDetaildistributionisstatisticallyexpectableinthesamesensethattheone-over-frequencymagnitudespectrumisstatisticallyexpectable.Thehistogramoforiginallumavaluesisnotstatisticallyexpectable:Itchangessignificantlyfromonevideosequencetoanotherandevenbetweenscenesofthesameprogram.

Figure7-TheSpatialDetailSignalDistributionisCompact,Symmetric,&Biphasic

EffectofHEVCCompressiononSpatialDetailCorrelationTheSpatialDetailsignalmightbethoughtofasthecondensedessenceoftheoriginalimage.Assuch,weexploredthepossibilitythatchangesintheSpatialDetailsignalthatresultfromcompressionmightprovetobeausefulindicatorofdistortionsandartifacts.

Weuseda10-bitbuildofx265(HEVC)tocompresseachofthetestsequencesatfivedifferentlevelsusingthe“constantquality”crfparameter(10,15,20,25,and30).Theinputtox265ineachcasewastheYCbCr4:2:210-bitversionoftheoriginalcontent.Theinternalx265compressedpixelformatwassetasYCbCr4:2:010-bittosimulatecable&payTVworkflows.TheresultingaveragebitratesareplottedinFigure8.WethendecodedeachframeofeachcompressedbitstreamtoYCbCr4:2:210-bitfordirectcomparisonwiththeinput.

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 12

Figure8-BitratesforHEVC-CompressedTestSequences

WediscoveredthatsimplecorrelationanalysisoftheSpatialDetailsignalsprovidesausefulmetric.Thecorrelationofthelumavaluesoftheuncompressedpixelarrays(horizontalaxis)andcorrespondingcompressedpixelarray(verticalaxis)areshownintheupperrowofFigure9forcrfvalues10(middle)and30(right).TheanalogousgraphsonthelowerrowareforthevaluesofthecorrespondingSpatialDetailsignals.Iftheuncompressedandcompressedvalueswereidenticalthedatapointswoulddescribeaperfectlineofunityslope.Differencesbetweentheuncompressedandcompresseddatacauseascatterabouttheline.Morecompresseddata(largercrfvalue)canbeexpectedtoresultinalargeramountofscatter.NotethoughthatthechangeinscatteringismorepronouncedfortheSpatialDetailsignalthantheoriginallumavalues.MorecompressioncausesthescatteroftheSpatialDetailvaluestobecomemoreglobular,becomingmorecompactalongthelineofperfectcorrelationandexpandingperpendiculartothatline.

Theamountofscatter–theamountofuncorrelation–isquantifiablebythecoefficientofdetermination,R2(pronounced“R-squared”),whichisastatisticalmeasureoftheamountofpredictabilityofonedatasetgivenanotherdataset.Inourcaseofsimplelinearregression,R2issimplythesquareofthePearsoncorrelationcoefficient.AnR2valueof1meansperfectlycorrelatedandavalueof0meansperfectlyuncorrelated.

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 13

Figure9-CorrelationofLumaandCorrespondingSpatialDetailSignals

R2valuesforallthetestsequencesateverycompressionlevelareplottedinFigure10.Fortheoriginallumavalues(right-handgraph),thevalueofR2changesonlyslightlybetweencrfvaluesof10and30eventhoughthebitratechangesbyapproximately2ordersofmagnitude(seeFigure8).ForthecorrespondingSpatialDetailsignal,thestoryisverydifferent(left-handgraph).ThevalueofR2changessignificantlyoverthesamerangeofcrfvaluesandcorrespondingbitrates.

Figure10-CorrelationValuesforAllTestSequences&HEVCCompressionLevels

Resultsfromwell-establishedvideoqualitymetricsforthesametestsequencesandcompressionlevelsareplottedinFigure11toprovideapointofcomparisonandreference.PSNRdisplaysgoodsensitivityovertheentirerange.MS-SSIMisalsosensitivetocompressionintherangethatcanbeexpectedincableandpayTVservice,butonlyoveraverytinyrestrictedrangeofvaluesfrom0.98to1outofafullrangeof0

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 14

to1.Incomparison,R2valuesforSpatialDetailrangesfrom0.4to1outofafullrangeof0to1.

Figure11-PSNRandMS-SSIMValuesforAllTestSequences&CompressionLevels

UsingSpatialDetailtoProbeBright&DarkFeaturesandTexturesTheSpatialDetailsignalcanbedecomposedintotwosubcomponents(Figure12)thatcanbeusedasguidesforselectivelyanalyzingperceptuallysignificantfeaturesandtextures.A“Sign”map(lowerleftinFigure12)oftheSpatialDetailsignalcanbecreatedsimplyasabinaryimageinwhicheachpixelissetto0ifthecorrespondingSpatialDetailpixelisnegativeandsetto1ifitispositive.TheSignmapwilltendtohaveanequalnumberof0’sand1’sbecauseofthestatisticallyexpectablesymmetricbiphasicdistributionofSpatialDetailvalues.A“Significance”map(lowerrightinFigure12)canbecreatedsimplyastheabsolutevalueoftheSpatialDetailsignal.BrightregionsoftheSignificancemapcorrespondtolargerabsolutevaluesoftheSpatialDetailsignal.NotethattheSignificancemaptendstohighlightedges,borders,andothertransitionswhichisin-linewiththinkingoftheSpatialDetailsignalasaresultofatrue2-dimensionalspatialdifferentiationasdiscussedabove.

Figure12-DecompositionofSpatialDetailintoaSignmapandaSignificancemap

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 15

TheSpatialDetailsignalcanalsobedecomposedasillustratedinFigure13toprovideaguideto“brightfeatures”,“darkfeatures”,and“textures”.LargepositivevaluesoftheSpatialDetailsignalcanbeusedtodefinethelocationofbrightfeatures.Largernegativevaluescanbesimilarlyusedtodefinethelocationofdarkfeatures.TheremainingsmallerpositiveandnegativevaluesoftheSpatialDetailsignalthusdefinetextures.Absolutethresholdscouldbeusedbutwefinditmoreusefultousegradedweightedfunctionssuchasbutnotlimitedtothefollowing:

𝑊"#$%&' 𝑥, 𝑦 =𝑆 𝑥, 𝑦

𝑆 𝑥, 𝑦 + 𝑆.𝑆 𝑥, 𝑦 > 0

𝑊12#3 𝑥, 𝑦 =𝑆 𝑥, 𝑦

𝑆 𝑥, 𝑦 + 𝑆.𝑆 𝑥, 𝑦 < 0

𝑊'56'7#5 𝑥, 𝑦 = 1 −𝑊"#$%&' 𝑥, 𝑦 −𝑊12#3 𝑥, 𝑦

where𝑊"#$%&' 𝑥, 𝑦 ,𝑊12#3 𝑥, 𝑦 , and𝑊'56'7#5 𝑥, 𝑦 arepixelarrayweightingmapshavingvaluesbetween0and1,and𝑆 𝑥, 𝑦 istheSpatialDetailsignalderivedfromtheuncompressedlumacomponent,and𝑆.isatuningparameterthatadjuststheboundarybetweenfeatureandtexture(equivalenttotheverticaldashedlinesinthetopcentergraphofFigure13).

TheimageinthemiddleofthelowerrowofFigure13wasobtainedbymultiplyingeachred,green,andbluecolorplaneby𝑊'56'7#5 𝑥, 𝑦 .Thelowerrightimageillustratingthebrightfeatureswascreatedthesameway,butwith𝑊"#$%&' 𝑥, 𝑦 .Thelowerleftwascreatedusing𝑊12#3 𝑥, 𝑦 +𝑊"#$%&' 𝑥, 𝑦 tovisualizeallfeatures.(TheweightingmapineachcasewascalculatedusingtheSpatialDetailsignalofthelumacomponent.)

Figure13-BrightFeatures,DarkFeatures,andTextures

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 16

Theproportionoftheimagethatmaybedescribedasbrightfeatures,darkfeatures,andtexturesmaybequantifiedusingformulaeofthetypebelowforanNxMsizedvideoframe:

𝑃"#$%&' =?@ABCDE 6,F

G,HI,J

KL;𝑃12#3 =

?MNAO 6,FG,HI,J

KL;𝑃'56'7#5 =

?EPIEQAP 6,FG,HI,J

KL

TexturesaccountforthemajorityofeachoftheHDRWCGtestsequencesthoughfeaturesplayarelativelylargerroleinfor“smith_hammering”and“carousel_fireworks”,asillustratedinFigure14.

Figure14-RelativeProportionsofBrightFeatures,DarkFeatures,andTextures

SpatialDetailCorrelationforHDRWCGFeaturesandTexturesBrightanddarkfeaturesandtexturesareparticularlyimportantinHDRWCGvideo.TheyarewhatmakeHDRpop.Weusedcorrelationanalysistoseeifthebrightfeatures,darkfeatures,ortexturesweresystematicallyaffectedbyHEVCcompressionpreferentially.

TheresultingR2valuesareplottedinFigure15.WefoundthatHEVCdidaparticularlygoodjobofpreservingboththebrightanddarkfeaturesevenatcompressionlevelsbeyondthatwhichwouldnormallybeusedincableandpayTVservices.Throughouttherangeofcompressionlevelswetested,theR2valuesforallfeaturesremainedabove0.9.Theresultsfortexturewerenotasgood.R2valuesfortexturedroppedbelow0.9evenforlightHEVCcompressionthusindicatingsignificantdistortion.ThesefindingswereconsistentacrossthetestsequencesthusindicatingasystematiccharacteristicofHEVCcompressionratherthanacontent-dependenteffect.

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 17

Figure15-SpatialDetailCorrelationforBright&DarkFeaturesandTextures

WeightedMean-SquaredErrorWealsoinvestigatedselectivedistortionforbright&darkfeaturesandtexturesusingweightedMean-SquaredError(MSE).Theweightingwasachievedbymultiplyingthesquareddifferencebetweentheuncompressedandcompressedvideoframedatabeforesummationoverallpixels(framesizeofNxM),asillustratedintheequationsbelow.

𝑀𝑆𝐸'T'2U =𝑌#5W 𝑥, 𝑦 − 𝑌'X' 𝑥, 𝑦

YK,L6,F

𝑁𝑀

𝑀𝑆𝐸'T'2U = 𝑀𝑆𝐸"#$%&' + 𝑀𝑆𝐸12#3 + 𝑀𝑆𝐸'56'7#5

𝑀𝑆𝐸"#$%&' =𝑊"#$%&' 𝑥, 𝑦 𝑌#5W 𝑥, 𝑦 − 𝑌'X' 𝑥, 𝑦

YK,L6,F

𝑁𝑀

Thevaluesof𝑀𝑆𝐸12#3and𝑀𝑆𝐸'56'7#5 maybecalculatedinasimilarmanner.TheresultingweightedMSEvaluesprovideinsightintotheproportionofthetotalMSEmaybeattributedtobright&darkfeaturesandtextures.ThesamemethodologymaybeappliedtobothlumaandchromaMSEswithappropriatescalingforthe4:2:2YCbCrformat.

WeightedMSEresultsfortheHDRWCGtestsequencesthatareplottedinFigure16demonstratethatthemajorityofthetotalMSEisattributabletothetexturecomponent.WefoundthisconclusiontobeconsistentacrossallHDRWCGtestsequencesforallcompressionlevelswetestedandthattheconclusionholdsforlumaandchroma.ThedominanceoftextureMSEismainlyaresultoftexturemakingupthelargestproportionofvideoframes(seeFigure14).

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 18

Figure16-WeightedMSEforBright&DarkFeaturesandTextures

Squared-ErrorDensityIntroductionofaSquared-ErrorDensity(SED)providesameansofselectivelyprobingdistortionforfeaturesandtextureswhileaccountingforeachone’srelativeprominenceinHDRWCGvideo.SEDmaybecalculatedforbright&darkfeatures,andtexturesaccordingtothefollowingequations:

𝑆𝐸𝐷"#$%&' =L\]@ABCDE^@ABCDE

;𝑆𝐸𝐷12#3 =L\]MNAO^MNAO

;𝑆𝐸𝐷'56'7#5 =L\]EPIEQAP

EPIEQAP

SEDisMSEdividedbythecorrespondingproportionalityoffeatureortexture.SEDthusaccountsforthefactthatfeaturestendtoberarerthetexture(seeFigure14).SEDmaybethoughtofasprovidingameasureofequitabilitybetweenfeaturesandtextures.Forexample,SEDcanprovideinsightintowhetherrarerfeaturesexperiencedisproportionatedistortioncomparedtotexture.

SEDresultsfortheHDRWCGtestsequencesareplottedinFigure17.Wefindsquared-errordensityforbrightanddarkfeaturesisrelativelymoreseverethanfortextures.ThisfindingisconsistentforallHDRWCGtestsequencesandcompressionlevelswetestedandholdsforlumaandchroma.

Figure17-Squared-ErrorDensityforBright&DarkFeaturesandTextures

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 19

CONCLUSIONWehavepresentedinthispaperasetofvideodistortionsmetricsthatmightprovetobeparticularlyusefulforHDRWCGvideo.Themainmotivatingprinciplewepresentedwasthe“SpatialDetail”signalthatweusedintwoways:1)asaproxyfortheoriginalimagedata;and2)asaguidetotheperceptuallyimportant“features”and“textures”inHDRWCGvideo.

TheSpatialDetailsignalisacondensedversionoftheoriginalimagethatpreservestherecognizabledetailsinanimagewhilediscountinglocalluminance.Itcanbethoughtofasatrue2-dimensionaldifferentialoftheoriginalimage.Itmayalsobeunderstoodintermsofthephaseinformationinanimageinconjunctionwiththestatisticallyunpredictableinformationinanimage.Fromapracticalstandpoint,itdoesn’treallymatterwhichtheoryoneprefers.Instead,animportantkeycharacteristicoftheSpatialDetailsignalisthatithasastatisticallystableandexpectablecompact,peaked,biphasic,andsymmetricdistributionofvaluesthatispreservedacrossawiderangeinimagesandvideo.Largervalues–positiveandnegative–formaconvenientguidetothekindsoffeaturespeopletendtofindsignificant.SpatialDetailvaluesnearerthezeromidpointofthedistributionformaconvenientguidetoimageregionsthatpeoplewouldtendtoclassifyastextural.SuchfeatureandtexturemapsprovideastableframeworkinwhichtoselectivelyinvestigatetheperceptualpotenthighlightsanddarkdetailsthatarethehallmarkofHDRWCGvideo.

WepresentedthreeHDRWCGvideodistortionmetricsinthispaper:

1. Forthefirstmetric,weusedSpatialDetailasaproxyfortheoriginalimageandshowedthatcorrelationbetweentheSpatialDetailsignalsoftheuncompressedandcompressedversionsofHDRWCGvideowassystematicallyaffectedbytheaggressivenessofHEVCcompression.BycombiningSpatialDetailcorrelationwithourfeatureandtextureassignmentmethods,weshowedthattexturecorrelationwasimpactedsignificantlymorethanfeaturecorrelation.SpatialDetailcorrelationhasseveraldistinctionswhencomparedtoestablishedvideoqualitymetrics.Itcanbeusedselectivelyonbright&darkfeaturesandontextures.Moreover,SpatialDetailvaluesareintherangeof0to1,whichismoreintuitivethantheunboundedPSNRscale,whilebeingamuchmoresensitiveindicatorthanMS-SSIMovertherangeofcompressionlevelstypicalofcableandpayTVoperations.

2. Forthesecondmetric,weusedSpatialDetailasaguideforbright&darkfeaturesandtexturetoselectivelyquantifytheMSEforeachlayerofimagedetail.WeshowedthattextureisthelargestcontributortooverallMSEmainly,becausetextureregionstypicallymakeupalargerproportionofanyimagethantherarerfeatureregions.

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 20

3. ThethirdmetricintroducedaSquared-ErrorDensity(SED)thatcompensatesfortherelativeproportionsoffeatureandtextureinanimagesoasassessdistortionsonamoreequalscale.WefoundthatSEDindicatesthatfeaturesexperiencedisproportionatedistortioncomparedtotexture.

Wehavedeliberatelyusedtheterm“videodistortion”insteadof“videoquality”throughoutthispaper.Themainreasonfordoingsoisthatthemetricsweproposedhavenotyetbeencomparedtosubjectivetestscoresandthusmaynotyetbeclaimedtobecalibratedsubjectivequalitymetrics.Also,itisnottheintentofthispapertolinkthemetricsweproposetosubjectiveassessment;thoughwemaydosoinlatterpublications.Rather,ourintentistoprovideeasytocalculatemetricsthatwehopecanprovideinsightduringthiscriticalperiodinourindustryasweworkthroughthetechnicalandcreativeissuesrelatedtoHDRandWCG.

ItisalsoworthhighlightingthattheSpatialDetailsignalandrelatedmetricsareeasytocalculateusingmodernsignalprocessingtechniquesinmodernprocessors.Thus,webelievethetechnicalbarriertoadoptionofthesemetricsislow.

Ourintentinthepaperistoprovideusefulandeasy-to-calculatemetricsthathavealowtechnicalbarriertoadoption.TheSpatialDetailsignalandrelatedmetricsweproposeareeasyenoughtocalculatethattheyarecandidatesforreal-timeHDRWCGvideoassessmentusingmodernsignalprocessingtechniquesinmodernprocessors.OurnextstepswillbetocontinuetoassesstheutilityofourHDRWCGmetricswiththehopethattheywillhelpMSOsnavigatekeytechnicalandcreativeissuesasHDRWCGvideoprogrammingemergesasthenextwaveofgreatsubscriberexperiences.

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 21

ABBREVIATIONSFFT2 2-dimensionalFastFourierTransformFSIM Feature-SimilarityIndexHDR HighDynamicRangeHEVC HighEfficiencyVideoCodingICTCP ICTCPcolorspaceIFFT2 Inverse2-dimensionFastFourierTransformMSE MeanSquareErrorMSO MultipleSystemsOperatorsMS-SSIM MultiscaleStructuralSimilarityPQ PerceptualQuantizerPSNR PeakSignal-to-NoiseRatioPU PerceptuallyUniformSDR StandardDynamicRangeSED Squared-ErrorDensitySSIM StructuralSimilarityYCbCr YCbCrcolorspaceVDP VisualDifferencePredictorVIF VisualInformationFidelityVQM VideoQualityMeasureYUV YUVcolorspaceWCG WideColorGamut

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 22

RELATEDREADINGS• ASystematicApproachtoVideoQualityAssessmentandBitratePlanning–In

thispaper,theauthorpresentsastreamlinedmethodofsettingoperationalvideoqualityandbandwidthusingeithersubjectiveorobjectivetesting,usingindividualgolden-eyesorfocusgroupsofanysize.ThedataandanalysisincludedareintendedtoaidinplanningvideoqualityandbandwidthresourcesacrossarangeofserviceofferingsfromOTTthroughUltraHD.

• EfficientContentProcessingforAdaptiveVideoDelivery–Thispaperprovidesanin-depthoverviewoftwoemergingtechnologies,dynamicprofileselectionandcooperativetranscoding,alongwithexperimentaldatademonstratingtheirpotentialforsubstantiallyreducingcontentprocessingrequirementsformultiscreenvideodelivery.

• MethodologiesforQoEMonitoringofIPVideoServices–ThispaperdiscussesthedifferencesbetweenQoEandQoSandbetweenQoEandvideoqualityandthencomparesdifferentmethodologiesforvideoqualityandQoEmonitoring.ItalsoincludesareviewofalternativesforembeddingQoEprobesintheend-to-endIPVideoarchitectureandtheirabilitytocollecttrueandeffectiveQoEinformation.

MEETOUREXPERT:SeanT.McCarthyDr.SeanMcCarthy,FellowoftheTechnicalStaff,bringsauniqueconvergenceofexpertiseinvideocompression,signalprocessing,andtheneurobiologyofhumanvisiontocontentdistributionatARRIS.Dr.McCarthyleadsadvancementsinstate-of-the-artofvideoprocessing,compressionandpracticalvisionscience.Previously,heheldsimilarresponsibilitiesasFellowoftheTechnicalStaffatMotorola,andasChiefScientistatbothModulusVideo,whichwasacquiredbyMotorola.Priortothat,Dr.McCarthyhadsimilarresponsibilitiesatViaSense,aUniversityofCalifornia,Berkeleyspin-offthatdevelopedcommercialapplicationsofthehumanvisualsystem.HeearnedaB.S.inphysicsfromRensselaerPolytechnic,andearnedhisPh.D.inbioengineeringjointlyatUniversityofCalifornia,BerkeleyandUniversityofCalifornia,SanFrancisco.

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 23

REFERENCES(1) Hanhart,P.,Kroshunov,P.,andEbrahimi,T.“Subjectiveevaluationofhigherdynamicrangevideo.”ProceedingsofSPIE-TheInternationalSocietyforOpticalEngineering,2014

(2) Wang,Z.,Bovik,A.C.,Sheikh,H.R.,andSimoncelli,E.P.“Imagequalityassessment:fromerrorvisibilitytostructuralsimilarity,”IEEETransactionsonImageProcessing,vol.13,no.4,pp.600–612,Apr.2004

(3) Hanhart,P.,Bernardo,M.V.,Korshunov,P.,andPereira,M.“HDRimagecompression:Anewchallengeforobjectivequalitymetrics.”inSixthInternationalWorkshoponQualityofMultimediaExperience(QoMEX),2014

(4) ITU-RBT.500-13,“Methodologyforthesubjectiveassessmentofthequalityoftelevisionpictures,”InternationalTelecommunicationUnion,Jan.2012

(5) ITU-TP.910,“Subjectivevideoqualityassessmentmethodsformultimediaapplications,”InternationalTelecommunicationUnion,April2008

(6) Winkler,S.DigitalVideoQuality:VisionModelsandMetrics,JohnWiley&Sons,Mar.2005

(7) VQEG,“Finalreportfromthevideoqualityexpertsgrouponthevalidationofobjectivemodelsofvideoqualityassessment,”Mar.2000.http://www.vqeg.org/.

(8) Wang,Z.andBovik,A.C.“Meansquarederror:loveitorleaveit?-Anewlookatsignalfidelitymeasures,”IEEESignalProcessingMagazine,vol.26,no.1,pp.98-117,Jan.2009

(9) Hanhart,P.,Bernado,M.V,Pereira,M.,Pinheiro,M.G.,andEbrahimi,T.“BenchmarkingofobjectivequalitymetricsforHDRimagequalityassessment.”EURASIPJ.Image&VideoProcessing,2015

(10) Wang,Z.,Simoncelli,E.P.,andBovik,A.“Multi-scalestructuralsimilarityforimagequalityassessment.”Proc.ofthe37thIEEEAsilomarConferenceonSignals,Systems,andComputers,2003

(11) Sheikh,H.R.andBovik,A.C.“Imageinformationandvisualquality,”IEEETransactionsonImageProcessing,vol.15,no.2,pp.430–444,Feb.2006

(12) Field,D.J.“Relationshipbetweenthestatisticsofnaturalimagesandtheresponsepropertiesofcorticalcells.”J.Opt.Soc.Am.A.Vol.4,No.12,1987

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 24

(13) Nill,N.B.andBouzas,B.H.“Objectiveimagequalitymeasurederivedfromdigitalimagepowerspectra.”OpticalEngineering,vol31,no.4,1992

(14) Liu,r.andLaganiere,R.“Ontheuseofphasecongruencytoevaluateimagesimilarity.”IEEEInternationalConferenceonAcousticsSpeechandSignalProcessingProceedings,2006

(15) Liu,r.andLaganiere,R.“Phasecongruencemeasurementforimagesimilarityassessment.”PatternRecognitionLetters,vol28,no.1,2007

(16) Kovesi,P.“ImageFeaturesfromPhaseCongruency.”inVidere:JournalofComputerVisionResearch,Vol1,No.3,TheMITPress,1999

(17) Kovesi,P.“Invariantmeasuresofimagefeaturesfromphaseinformation.”Thesis(Ph.D.)Dept.ofComputerScience.UniversityofWesternAustralia,1996

(18) Morrone,M.C.andOwens,R.A.“Featuredetectionfromlocalenergy.”PatternRecognition.”Lett.,303–313,1987

(19) Morrone,M.C.andBurr,D.C.“Featuredetectioninhumanvision:Aphase-dependentenergymodel.”Proc.RoyalSoc.OfLondon,SeriesB,BiologicalSciences,vol.235,no.1280,1988

(20) Zhang,L.,Zhang,L.,Mou,X.,andZhang,D.“FSIM:Afeaturesimilarityindexforimagequalityassessment.”IEEETrans.ImageProcess.vol20,no.8,2011

(21) Mantiuk,R.,Daly,S.,Myszkowski,K.,andSeidel,H.-P.“Predictingvisibledifferencesinhighdynamicrangeimages:modelanditscalibration.”SPIEHumanVisionandElectronicImagingX,vol.5666.,2005

(22) Daly,S.J.“Visibledifferencespredictor:analgorithmfortheassessmentofimagefidelity”SPIEHumanVision,VisualProcessing,andDigitalDisplayIII,vol.1666.,1992

(23) Mantiuk,R.,Kim,K.J.,Rempel,A.G.,andHeidrich,W.“HDR-VDP-2:Acalibratedvisualmetricforvisibilityandqualitypredictionsinallluminanceconditions.”ACMTrans.Graph.30(4),40:1–40:14,2011

(24) Narwaria,M.,Mantiuk,R.K.,PerreiraDaSilva,M.,andLeCallet,P.“HDR-VDP-2.2:acalibratedmethodforobjectivequalitypredictionofhigh-dynamicrangeandstandardimages.”J.Electron.Imaging.24(1),010501,2015

(25) Aydin,T.O.,Mantiuk,R.,andSeidelH.-P.“Extendingqualitymetricstofullluminancerangeimages.”HumanVisionandElectronicImagingXIII.EditedbyRogowitz,BerniceE.;Pappas,ThrasyvoulosN.ProceedingsoftheSPIE,Volume6806,2008

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 25

(26) Miller,S.,Nezamabadi,M.,andDaly,S.,“PerceptualSignalCodingforMoreEfficientUsageofBitCodes.”SMPTEMotionImagingJournal,2013

(27) Valenzise,G.,DeSimone,F.,Lauga,P.,andDufaux,F.“PerformanceevaluationofobjectivequalitymetricsforHDRimagecompression.”Proc.SPIE9217,ApplicationsofDigitalImageProcessingXXXVII,2014

(28) Mantel,C.,Ferchiu,S.C.,Forchhammer,S.“ComparingsubjectiveandobjectivequalityassessmentofHDRimagescompressedwithJPEG-XT.”in16thInternationalWorkshoponMultimediaSignalProcessing(MMSP),IEEE,2014

(29) Rerabek,M.,Hanhart,P.,Korshunov,P.,andEbrahimi,T.“SubjectiveandobjectiveevaluationofHDRcompression.”InternationalWorkshoponVideoProcessingandQualityMetricsforConsumerElectronics-VPQM,Chandler,Arizona,USA.February2015

(30) McCarthy,S.T.andOwen,W.G.“ApparatusandMethodsforImageandSignalProcessing”.USPat.6014468(2000).USPat.6360021(2002),USPat.7046852(2006),1998

(31) McCarthy,S.,“ABiologicalFrameworkforPerceptualVideoProcessingandCompression,”SMPTEMot.Imag.J.,119(8):24-32,Nov/Dec.2012

(32) McCarthy,S.T.“Theoryandpracticeofperceptualvideoprocessinginbroadcastencodersforcable,IPTV,satellite,andinternetdistribution.”Proc.SPIE9014,HumanVisionandElectronicImagingXIX,2014

(33) Froehlich,J.,etal.“HdM-HDR-2014Project,”http://www.hdm-stuttgart.de/~froehlichj/hdm-hdr-2014

(34) Froehlich,J.,Grandinetti,S.,Eberhardt,B.,Walter,S.,Schillin,A.,andBrendel,H.“CreatingcinematicwidegamutHDR-videofortheevaluationoftonemappingoperatorsandHDR-displays.”Proc.SPIE9023,DigitalPhotographyX,2014

(35) ITU-RBT.2020“Parametervaluesforultra-highdefinitiontelevisionsystemsforproductionandinternationalprogrammeexchange."InternationalTelecommunicationUnion,2012

(36) TheMathworks.www.mathworks.com

(37) ffmpeg.www.ffmpeg.org

(38) x265.www.x265.org

(39) ITU-RBT.2100.“Imageparametervaluesforhighdynamicrangetelevisionforuseinproductionandinternationalprogrammeexchange.”2016

Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 26

(40) ReportITU-RBT.2390-0“Highdynamicrangetelevisionproductionandinternationalprogrammeexchange.”InternationalTelecommunicationUnion,2016