SETTINGVIDEOQUALITY&PERFORMANCETARGETSFORHDRANDWCGVIDEOSERVICES
SEANT.MCCARTHY
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 2
TABLEOFCONTENTSINTRODUCTION.............................................................................................3QuantifyingHDRWCGVideoQuality&Distortions.......................................................3
ThePerformanceofExistingHDRVideoQualityMetrics...............................................4
BalancingPerformanceandComplexity.........................................................................5
CHARACTERISTICSOFHDRWCGVIDEO........................................................6TestSequences&Preparation.......................................................................................6
RepresentingImagesinTermsofSpatialFrequency......................................................6
ExpectableStatisticsofComplexImages.......................................................................7
PROPOSEDHDRWCGVIDEODISTORTIONALGORITHM...............................8SpatialDetail...................................................................................................................8
EffectofHEVCCompressiononSpatialDetailCorrelation..........................................11
UsingSpatialDetailtoProbeBright&DarkFeaturesandTextures.............................14
SpatialDetailCorrelationforHDRWCGFeaturesandTextures..................................16
WeightedMean-SquaredError....................................................................................17
Squared-ErrorDensity..................................................................................................18
CONCLUSION...............................................................................................19ABBREVIATIONS...........................................................................................21RELATEDREADINGS.....................................................................................22REFERENCES................................................................................................23
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 3
INTRODUCTIONHighDynamicRange(HDR)andWideColorGamut(WCG)canhaveabigpositiveimpactonaviewerbycreatingamoreconvincingandcompellingsenseoflightthanhaseverbeforebeenpossibleintelevision.Arecentscientificstudy1withprofessional-qualityStandardDynamicRange(SDR)andHDRvideosfoundthatviewerspreferHDRoverSDRbyalargemargin.Moreover,thestudyalsoshowedthatthemarginofpreferenceforHDRincreasedwithincreasingpeakluminance.
Whathappensthoughtoaviewer’squalityofexperiencewhenpristinehighqualityHDRcontentiscompressedfordistribution?WhathappenswhenHDRWCGcontentisconvertedtoSDRcontenttosupportlegacydisplaysandconsumerset-topboxes?DodistortionsandcompressionartifactsbecomemorenoticeableinHDR?DoesprocessedHDRlosesomeofitssparkleandbecomelessdiscerniblefromordinarySDR?
Videoqualityiseasytorecognizebyeye,butputtinganumberonvideoqualityisoftenmoreproblematic.ForHDR&WCGtheproblemisevenharder.HDR&WCGaresoperceptuallypotentbecauseevenrelativelyinfrequentfeaturessuchasspecularreflectionsandsaturatedcolorscanengageaviewer’sattentionfully.Yet,well-knownvideo-qualityscoringmethods,suchaspeaksignal-to-noiseratio(PSNR)andtheStructuralSIMilaritymetric2(SSIM),couldleadtowrongconclusionswhenappliedtotheperceptualoutliersinHDRWCGvideo.Withoutgoodvideo-qualitymetrics,cableoperatorscannotmakeinformeddecisionswhensettingbitrateandvideo-qualityperformancetargets,norwhenchoosingtechnologypartnersforHDRWCGservices.
WeneedawayofquantifyingdistortionsintroducedduringHDRWCGvideoprocessingthattakesintoaccountthewideluminancerangeofHDRvideoaswellasthelocalizedhighlights,deepdarks,andsaturatedcolorsthatgiveHDRWCGitsspecialappeal3.
Thispaperintroduceseasy-to-calculatequantitativemethodstoprovidecableoperatorswithvideo-qualitydatathatcanbeusedtomakeoperational,technological,andproductdecisions.Specifically,itpresentsmethodstoreportthelevelofoveralldistortionsinprocessedvideoaswellasthespecificdistortionsassociatedwithperceptuallyimportantbright&darkHDRfeaturesandtextureswithrespecttobothlumaandchromacomponents.Thepaper’sobjectiveistoshowdataandanalysisthatillustrateshowquantifyingHDRWCGvideodistortioncanbemadeaccurate,actionable,andpractical,particularlywhenMSOsconsiderthevarioustrade-offsbetweenbandwidth,technologyoptions,andtheviewer’sexperience.
QuantifyingHDRWCGVideoQuality&DistortionsThebestwaytoquantifyvideoqualityandviewerpreferenceistoperformsubjectivetestingusingestablishedtechniquesandexistinginternationalstandardssuchasITU-R
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 4
BT.5004andITU-TP.9105;butsubjectivetestingistooslowtobepracticalinmostsituations.Instead,anumberofobjectivevideoqualityassessmenttechniquesandmetricshavebeendevelopedoverthedecades6.Objectivevideoqualityassessmentreliesoncomputeralgorithmsthatcanbeinsertedintoproductionanddistributionworkflowstoprovideactionableinformation.Somevideoqualityalgorithms,suchasPSNR,areverysimple,butdonotcorrelatewellwithsubjectivescores7,8.Othersareverysophisticatedandincludemodelsofthehumanvisualsystem.Suchmetricsdoabetterjobofpredictingsubjectiveresults,butcansufferfromcomputationalcomplexitythatlimitstheiruniversalusefulness9.Stillsomeothervideoqualitymetrics,suchasSSIMandmultiscaleMS-SSIM10,haveemergedthatstrikeagoodandusefulbalancebetweencomplexityandabilitytopredicthumanopinionswithreasonableaccuracy.
Anotherimportantclassofvideoqualitymetricsanalyzesprimarilythesignalcharacteristicsofimages,thoughtheyoftenalsoincludesomeaspectofthehumanvisualsystem.TheVIFmetricdevelopedbySheikhandBovik11,forexample,incorporatesthestatisticsofnaturalscenes12.NillandBouzas13developedanobjectivevideoqualitymetricbasedontheapproximateinvarianceofthepowerspectraimages.Lui&Laganiere14,15developedamethodofusingphasecongruencytomeasureimagesimilarityrelatedtoworkbyKovesi16,17andbasedontheproposalbyMorrone&Owens18andMorrone&Burr19andthatperceptuallysignificantfeaturessuchaslinesandedgesarethefeaturesinanimagewherethespatialfrequencycomponentscomeintophasewitheachother.Morerecently,Zhangetal.20leveragedtheconceptofphasecongruencytodevelopFSIM,afeaturesimilaritymetric.
Themetricweproposeinthispaperfallsinwiththeabovegroupofmetrics.Itsharesthesamemindspaceinthatitreferencesstatisticallyexpectablespatialfrequencystatisticsandthesignificanceofphaseinformationinanimage;butalsoitdiffersinseveralimportantaspects.Themetricweproposedoesnotrelyonphasecongruencybutratherona“SpatialDetail”signalthatcanbethoughtofasacombinationofthetruephaseinformationinanimageandthestatisticallyunpredictableinformationinanyparticularimage.The“SpatialDetail”signalcanbethoughtofasthecondensedessenceofanimagethathasthetwinadvantagesofbeingveryeasytocalculateandofprovidingaguidetothebrightanddarkfeaturesandtexturesthatgiveHDRWCGitsspecialappeal.
ThePerformanceofExistingHDRVideoQualityMetricsItwouldbesimpleifwecouldusetheSDRobjectivevideoqualitymetricswehavecometoknowsowelltoquantifyHDRvideoqualityalso.ItturnsoutthatobjectivevideoqualityassessmentforHDRisnotsimple.HDRvideoqualityassessmentneedseithernewalgorithmsandmetricsoranewmoreperceptuallymeaningfulwayofrepresentingimagedata.Perhapsbothwillbeneeded.
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 5
Hanhart,etal.1,recentlyreportedastudyofobjectivevideoqualitymetricsforHDRimages.Theylookedattheaccuracy,monotonicity,andconsistencyofalargenumberofbothlegacySDRandnewerHDR-specificmetrics21-24withrespecttoeachmetric’spredictionofsubjectivevideoqualityscores.TheyfoundthatmetricssuchasHDR-VDP-223andHDR-VQM24thatweredesignedspecificallyforHDRcontentwerebest.
Interestingly,Hanhartetal.alsofoundthattheperformanceofmostfull-referencemetrics,includingPSRNandSSIM,wasimprovedwhentheywereappliedtononlinearperceptuallytransformedluminancedata(PU25andPQ26)insteadoflinearluminancedata.AsimilarconclusionwasreportedearlierbyValenziseetal.27whousedaperceptuallyuniform“PUtransform”developedbyAydinetal.25toassesscompressedHDRimages.TheyfoundthatPU-basedPSNRandSSIMperformedaswellandsometimesbetterthanthemorecomputationallydemandingHDR-VDP21algorithm.AnotherstudybyManteletal.28alsoreportedthatperceptuallinearizationinfluencedtheperformanceofobjectivemetrics,thoughinthisstudyperceptuallinearizationdidnotalwaysimproveperformance.Rerabeketal..29extendedthestudyofobjectivemetricsbeyondstillimagestoHDRvideosequencesandfoundthatperceptuallyweightedvariantsofPSNR,SSIM,MSE,andVIFcorrelatedwellwithsubjectivescores,thoughHDR-VDP-2wasfoundtobethebestperformerstatistically.
BalancingPerformanceandComplexityObjectivevideoqualityalgorithmsshouldbeassimpleaspossibleandnosimpler.Complexmodelsofhumanvisionareimportantandhavetheirplace,butcanalsobecometoocumbersometobepracticallydeployedinproductionanddistributionofvideoprograms.Ontheotherhand,simplerfidelitymetricssuchasPSNR,SSIM,andMS-SSIMmightbesettingthebartoolowevenwithperceptuallylinearizedimagedata.
ThispaperproposesnewHDRWCGvideodistortionmetricsandanalgorithmthatisintendedtobesimple,fast,andprovideactionabledatatomonitorandimproveeverydayvideooperations.
ThevideodistortionassessmentmethodwepresentleveragesaframeworkofbiologicallyinspiredimageandvideoprocessingdevelopedbyMcCarthy&Owen30,31basedonstudiesofthevertebrateretinaandtheexpectablestatisticsofnaturalscenes.Thisbio-inspiredframeworkhasbeenleveragedpreviouslytodevelopaperceptualpre-processorusedinprofessionbroadcastencoders32tomakevideomorecompressiblewhileminimizingintroducedartifacts.Thedetailsofthetheoryarebeyondthescopeofthepaper,buttheapplicableelementsofthetheorycanperhapsbestbeexplainedbyconsideringvideointermsofspatialfrequency(seeFigure2).
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 6
CHARACTERISTICSOFHDRWCGVIDEOTestSequences&PreparationInthisstudy,weusedtheHDRWCGtestsequencesshowninFigure1.Thesesequenceswerecreatedbythe“HdM-HDR-2014Project”33,34toprovideprofessionalqualitycinematicwidegamutHDRvideofortheevaluationoftonemappingoperatorsandHDRdisplays.Allclipsare1920x1080p24andcolorgradedforRec.2020primaries&0.005-4000cd/m2luminance.TosimulatecableandpayTVscenarios,weconvertedtheoriginalcolorgradedframes(RGB48bitsperpixelTIFFfiles)toYCbCrv210format(4:2:210bit)usingtheequationsdefinedinITU-RBT.202035.AllvideoprocessingandanalysiswasperformedusingMatlab36,ffmpeg37,andx26538.
Figure1-HDRWCGTestSequencesUsedinthisStudy
RepresentingImagesinTermsofSpatialFrequencyAnimageisnormallythoughtofasa2-dimensionalarrayofpixelswitheachpixelbeingrepresentedbyred,green,andbluevalues(RGB)oralumaand2chromachannels(forexample,YUV,YCbCr,andmorerecentlyICTCP).Animagecanalsoberepresentedasa2-dimensionalarrayofspatial-frequencycomponentsasillustratedinFigure2.Thevisualpixel-basedimageandthespatial-frequencyrepresentationofthevisualimageareinterchangeablemathematically.Theyhaveidenticalinformation,justorganizeddifferently.
Figure2-RepresentationofaVideoFrameinTermsofSpatialFrequency
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 7
Spatial-frequencydatacanbeobtainedfromanimagepixelarraybyperforminga2-dimensionalFastFourierTransform(FFT2).Thepixelarraycanberecoveredbyperforminga2-dimensionalInverseFastFourierTransform(IFFT2).FFT2andIFFT2arewellknownsignalprocessingoperationsthatcanbecalculatedquicklyinmodernprocessors.
Inthespatialfrequencydomain,theinformationinanimageisrepresentedasa2-dimensionalarraycomplexnumbers;orequivalentlyasthecombinationofareal-valued2-dmagnitudespectrumandareal-valued2-dphasespectrum.(NotethatthelogofthemagnitudespectrumisshowninFigure2toaidvisualization.ThehorizontalandverticalfrequencyaxesareshownrelativetothecorrespondingNyquistfrequency(±1).)
Figure3-ThePhaseSpectrumTypicallyContainsMostoftheDetailsofanImage
Thephasespectrumcontainsmostofthespecificdetailsontheimage,asillustratedinFigure3.Onewaytothinkofthephasespectrumisthatitprovidesinformationonhowthevariousspatialfrequenciesinteracttocreatethefeaturesanddetailswerecognizeinimages18,19.Themagnitudespectrumtypicallycarrieslittleuniqueidentifyinginformationaboutanimage.Instead,itprovidesinformationonhowmuchoftheoverallvariationwithinthevisual(pixel-based)imagecanbeattributedtoaparticularspatialfrequency.
ExpectableStatisticsofComplexImagesImagesofnaturalsceneshaveaninterestingstatisticalproperty:Theyhavespatial-frequencymagnitudespectrathattendtofalloffwithincreasingspatialfrequencyinproportiontotheinverseofspatialfrequency12.Themagnitudespectraofindividualimagescanvarysignificantly;butasanensemble-averagestatisticalexpectation,itcanbesaidthat“themagnitudespectraofimagesofnaturalscenesfalloffasone-over-
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 8
spatial-frequency.”Thisstatementappliestobothhorizontalandverticalspatialfrequencies.
Figure4-Illustrationof“One-Over-Spatial-Frequency”MagnitudeSpectra
Figure4demonstratesthatindividualframesoftheHDRWCGtestsequencesusedinthisstudygenerallyadheretothe“one-over-spatial-frequency”statisticalexpectation.Theplotsalongthebottomofthefigureshowthevaluesofthemagnitudespectrumalongtheprincipalhorizontal(orange)andvertical(blue)axescorrespondingtothehorizontal(orange)andvertical(blue)arrowsinthemiddlerowofthefigure.
Itisworthnotingthattheexpectablestatisticsof“natural-scene”imagesarenotlimitedtopicturesofgrassandtreesandthelike.Anyvisuallycompleximageofa3-dimensionalenvironmenttendstohavetheone-over-frequencycharacteristic,thoughman-madeenvironmentstendtohavestrongerverticalandhorizontalbiasthanunalteredlandscape.Theone-over-frequencycharacteristiccanalsobethoughtofasasignatureofscale-invariance,whichreferstothewayinwhichsmallimagedetailsandlargeimagedetailsaredistributed.Imagesoftextandsimplegraphicsdonottendtohaveone-over-frequencymagnitudespectra.
PROPOSEDHDRWCGVIDEODISTORTIONALGORITHMSpatialDetailHDRisallaboutpreservingspatialdetail.Itisnotaboutbrighterpictures39,40,oratleastitshouldnotbe.ThewiderluminancerangeencodedbyHDRenablescrispspatialdetail
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 9
indarkregionsandbrighthighlightstoplayaroleinstorytellingthatisnotpossibleotherwise.Similarly,WCGisallaboutenablingcolorfulnessofspatialdetails.
Whatis“spatialdetail?”Weknowitwhenweseeit;butifwecan’tmeasureitquantitativelywecan’tmanageitsystematically.
Weproposethat“spatialdetail”canbequantifiedasthephaseinformationinanimagecombinedwiththestatisticallyunexpectablevariationsinthemagnitudespectruminformation.
Figure5-MethodofCalculatingtheSpatialDetailSignal
OurmethodofcreatingaSpatialDetailsignalisillustratedinFigure5.First,themagnitudeandphasespectraarecalculatedfromtheimagepixelarray(onlythelumachannelisshowninFigure5,butthemethodologymayalsobeappliedtothechromachannelor,alternatively,tothered,green,andbluechannels.)Next,apredeterminedarchetypeofthestatisticallyexpectableone-over-frequencymagnitudespectrumisdividedintotheactualmagnitudespectrumtoproduceastatisticallyweightedmagnitudespectrum.Third,thestatisticallyweightedmagnitudespectrumiscombinedwiththeactualphasespectrum.Finally,a2-dimensionalInverseFastFourierTransformisappliedtoproduceapixelarraythatwecalltheSpatialDetailsignal(seeFigure6).
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 10
Figure6-EnlargedViewoftheSpatialDetailSignalfortheLumaComponent
TheSpatialDetailsignalcanbethoughtofastheresultofa“whitening”process.However,atruewhiteningisasignalprocessingoperationthatresultsinexactlyequalmagnitudevaluesatallfrequencies.ThephaseimageshowninFigure3istheresultofatruewhiteningprocess.ItisperhapsmoreusefulandaccuratetothinkoftheSpatialDetailastheresultof“statisticallyexpectablewhitening”thatcontainstheresultofatruewhitening(thephaseimage)filteredbythestatisticallyunexpectablemodulationsofthemagnitudespectrum.Thedistinctionmightseemnuanced,yetthedifferencehaspracticalbenefits.Whereasthephaseimage(Figure3)isroughand“noisy”inawaythatobscurestherecognizabledetailsinanimage,theSpatialDetailsignal(Figure6)isasmoothlyvaryingmorerecognizabledualoftheoriginalimage.
TheSpatialDetailsignalmayalsobethoughtofastheresultofatrue2-dimensionaldifferentiationoftheimagepixelarray.TheSpatialDetailsignalisobtainedbydividingtheactualmagnitudespectrumbyaone-over-frequencyspectrum,whichisequivalenttomultiplyingtheactualmagnitudespectrumbyfrequency.Multiplicationbyfrequencyinthefrequencydomainisequivalenttodifferentiationinthepixeldomain.
ThedifferentiationcharacteristicoftheSpatialDetailisapparentinFigure7.Thelumavaluesoftheoriginalpixelarray(A)alongthemidline(dashedline)areplottedintheuppermiddlegraph(C).Thehistogramoftheallthelumavaluesoftheoriginalpixelarrayareplottedintheupperrightgraph(E).ThecorrespondingSpatialDetailsignal(B)valuesalongthemidlineareplottedinthelowermiddlegraph(D).ThehistogramofthealltheSpatialDetailvaluesareplottedinthelowerrightgraph(F).NotethattheSpatial
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 11
Detailvaluestendtoclusternearzeroanddeviatesignificantlyfromthezerolineonlywheretheoriginallumavalueschangesignificantly.
NotealsothattheSpatialDetailhistogramiscenteredonzeroandissymmetric,biphasic,andformsacompactpeakeddistribution.Conversely,theoriginallumavaluesarespreadout.ThesignificanceofthisdistinctionisthatthedistributionofSpatialDetailvaluesispreservedacrossimages.Thewidthofthehistogramchangesmoderatelyfromonevideosequencetoanotherbutretainsthestereotypicalcompact,peaked,biphasic,andsymmetricshape.Inotherwords,theSpatialDetaildistributionisstatisticallyexpectableinthesamesensethattheone-over-frequencymagnitudespectrumisstatisticallyexpectable.Thehistogramoforiginallumavaluesisnotstatisticallyexpectable:Itchangessignificantlyfromonevideosequencetoanotherandevenbetweenscenesofthesameprogram.
Figure7-TheSpatialDetailSignalDistributionisCompact,Symmetric,&Biphasic
EffectofHEVCCompressiononSpatialDetailCorrelationTheSpatialDetailsignalmightbethoughtofasthecondensedessenceoftheoriginalimage.Assuch,weexploredthepossibilitythatchangesintheSpatialDetailsignalthatresultfromcompressionmightprovetobeausefulindicatorofdistortionsandartifacts.
Weuseda10-bitbuildofx265(HEVC)tocompresseachofthetestsequencesatfivedifferentlevelsusingthe“constantquality”crfparameter(10,15,20,25,and30).Theinputtox265ineachcasewastheYCbCr4:2:210-bitversionoftheoriginalcontent.Theinternalx265compressedpixelformatwassetasYCbCr4:2:010-bittosimulatecable&payTVworkflows.TheresultingaveragebitratesareplottedinFigure8.WethendecodedeachframeofeachcompressedbitstreamtoYCbCr4:2:210-bitfordirectcomparisonwiththeinput.
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 12
Figure8-BitratesforHEVC-CompressedTestSequences
WediscoveredthatsimplecorrelationanalysisoftheSpatialDetailsignalsprovidesausefulmetric.Thecorrelationofthelumavaluesoftheuncompressedpixelarrays(horizontalaxis)andcorrespondingcompressedpixelarray(verticalaxis)areshownintheupperrowofFigure9forcrfvalues10(middle)and30(right).TheanalogousgraphsonthelowerrowareforthevaluesofthecorrespondingSpatialDetailsignals.Iftheuncompressedandcompressedvalueswereidenticalthedatapointswoulddescribeaperfectlineofunityslope.Differencesbetweentheuncompressedandcompresseddatacauseascatterabouttheline.Morecompresseddata(largercrfvalue)canbeexpectedtoresultinalargeramountofscatter.NotethoughthatthechangeinscatteringismorepronouncedfortheSpatialDetailsignalthantheoriginallumavalues.MorecompressioncausesthescatteroftheSpatialDetailvaluestobecomemoreglobular,becomingmorecompactalongthelineofperfectcorrelationandexpandingperpendiculartothatline.
Theamountofscatter–theamountofuncorrelation–isquantifiablebythecoefficientofdetermination,R2(pronounced“R-squared”),whichisastatisticalmeasureoftheamountofpredictabilityofonedatasetgivenanotherdataset.Inourcaseofsimplelinearregression,R2issimplythesquareofthePearsoncorrelationcoefficient.AnR2valueof1meansperfectlycorrelatedandavalueof0meansperfectlyuncorrelated.
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 13
Figure9-CorrelationofLumaandCorrespondingSpatialDetailSignals
R2valuesforallthetestsequencesateverycompressionlevelareplottedinFigure10.Fortheoriginallumavalues(right-handgraph),thevalueofR2changesonlyslightlybetweencrfvaluesof10and30eventhoughthebitratechangesbyapproximately2ordersofmagnitude(seeFigure8).ForthecorrespondingSpatialDetailsignal,thestoryisverydifferent(left-handgraph).ThevalueofR2changessignificantlyoverthesamerangeofcrfvaluesandcorrespondingbitrates.
Figure10-CorrelationValuesforAllTestSequences&HEVCCompressionLevels
Resultsfromwell-establishedvideoqualitymetricsforthesametestsequencesandcompressionlevelsareplottedinFigure11toprovideapointofcomparisonandreference.PSNRdisplaysgoodsensitivityovertheentirerange.MS-SSIMisalsosensitivetocompressionintherangethatcanbeexpectedincableandpayTVservice,butonlyoveraverytinyrestrictedrangeofvaluesfrom0.98to1outofafullrangeof0
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 14
to1.Incomparison,R2valuesforSpatialDetailrangesfrom0.4to1outofafullrangeof0to1.
Figure11-PSNRandMS-SSIMValuesforAllTestSequences&CompressionLevels
UsingSpatialDetailtoProbeBright&DarkFeaturesandTexturesTheSpatialDetailsignalcanbedecomposedintotwosubcomponents(Figure12)thatcanbeusedasguidesforselectivelyanalyzingperceptuallysignificantfeaturesandtextures.A“Sign”map(lowerleftinFigure12)oftheSpatialDetailsignalcanbecreatedsimplyasabinaryimageinwhicheachpixelissetto0ifthecorrespondingSpatialDetailpixelisnegativeandsetto1ifitispositive.TheSignmapwilltendtohaveanequalnumberof0’sand1’sbecauseofthestatisticallyexpectablesymmetricbiphasicdistributionofSpatialDetailvalues.A“Significance”map(lowerrightinFigure12)canbecreatedsimplyastheabsolutevalueoftheSpatialDetailsignal.BrightregionsoftheSignificancemapcorrespondtolargerabsolutevaluesoftheSpatialDetailsignal.NotethattheSignificancemaptendstohighlightedges,borders,andothertransitionswhichisin-linewiththinkingoftheSpatialDetailsignalasaresultofatrue2-dimensionalspatialdifferentiationasdiscussedabove.
Figure12-DecompositionofSpatialDetailintoaSignmapandaSignificancemap
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 15
TheSpatialDetailsignalcanalsobedecomposedasillustratedinFigure13toprovideaguideto“brightfeatures”,“darkfeatures”,and“textures”.LargepositivevaluesoftheSpatialDetailsignalcanbeusedtodefinethelocationofbrightfeatures.Largernegativevaluescanbesimilarlyusedtodefinethelocationofdarkfeatures.TheremainingsmallerpositiveandnegativevaluesoftheSpatialDetailsignalthusdefinetextures.Absolutethresholdscouldbeusedbutwefinditmoreusefultousegradedweightedfunctionssuchasbutnotlimitedtothefollowing:
𝑊"#$%&' 𝑥, 𝑦 =𝑆 𝑥, 𝑦
𝑆 𝑥, 𝑦 + 𝑆.𝑆 𝑥, 𝑦 > 0
𝑊12#3 𝑥, 𝑦 =𝑆 𝑥, 𝑦
𝑆 𝑥, 𝑦 + 𝑆.𝑆 𝑥, 𝑦 < 0
𝑊'56'7#5 𝑥, 𝑦 = 1 −𝑊"#$%&' 𝑥, 𝑦 −𝑊12#3 𝑥, 𝑦
where𝑊"#$%&' 𝑥, 𝑦 ,𝑊12#3 𝑥, 𝑦 , and𝑊'56'7#5 𝑥, 𝑦 arepixelarrayweightingmapshavingvaluesbetween0and1,and𝑆 𝑥, 𝑦 istheSpatialDetailsignalderivedfromtheuncompressedlumacomponent,and𝑆.isatuningparameterthatadjuststheboundarybetweenfeatureandtexture(equivalenttotheverticaldashedlinesinthetopcentergraphofFigure13).
TheimageinthemiddleofthelowerrowofFigure13wasobtainedbymultiplyingeachred,green,andbluecolorplaneby𝑊'56'7#5 𝑥, 𝑦 .Thelowerrightimageillustratingthebrightfeatureswascreatedthesameway,butwith𝑊"#$%&' 𝑥, 𝑦 .Thelowerleftwascreatedusing𝑊12#3 𝑥, 𝑦 +𝑊"#$%&' 𝑥, 𝑦 tovisualizeallfeatures.(TheweightingmapineachcasewascalculatedusingtheSpatialDetailsignalofthelumacomponent.)
Figure13-BrightFeatures,DarkFeatures,andTextures
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 16
Theproportionoftheimagethatmaybedescribedasbrightfeatures,darkfeatures,andtexturesmaybequantifiedusingformulaeofthetypebelowforanNxMsizedvideoframe:
𝑃"#$%&' =?@ABCDE 6,F
G,HI,J
KL;𝑃12#3 =
?MNAO 6,FG,HI,J
KL;𝑃'56'7#5 =
?EPIEQAP 6,FG,HI,J
KL
TexturesaccountforthemajorityofeachoftheHDRWCGtestsequencesthoughfeaturesplayarelativelylargerroleinfor“smith_hammering”and“carousel_fireworks”,asillustratedinFigure14.
Figure14-RelativeProportionsofBrightFeatures,DarkFeatures,andTextures
SpatialDetailCorrelationforHDRWCGFeaturesandTexturesBrightanddarkfeaturesandtexturesareparticularlyimportantinHDRWCGvideo.TheyarewhatmakeHDRpop.Weusedcorrelationanalysistoseeifthebrightfeatures,darkfeatures,ortexturesweresystematicallyaffectedbyHEVCcompressionpreferentially.
TheresultingR2valuesareplottedinFigure15.WefoundthatHEVCdidaparticularlygoodjobofpreservingboththebrightanddarkfeaturesevenatcompressionlevelsbeyondthatwhichwouldnormallybeusedincableandpayTVservices.Throughouttherangeofcompressionlevelswetested,theR2valuesforallfeaturesremainedabove0.9.Theresultsfortexturewerenotasgood.R2valuesfortexturedroppedbelow0.9evenforlightHEVCcompressionthusindicatingsignificantdistortion.ThesefindingswereconsistentacrossthetestsequencesthusindicatingasystematiccharacteristicofHEVCcompressionratherthanacontent-dependenteffect.
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 17
Figure15-SpatialDetailCorrelationforBright&DarkFeaturesandTextures
WeightedMean-SquaredErrorWealsoinvestigatedselectivedistortionforbright&darkfeaturesandtexturesusingweightedMean-SquaredError(MSE).Theweightingwasachievedbymultiplyingthesquareddifferencebetweentheuncompressedandcompressedvideoframedatabeforesummationoverallpixels(framesizeofNxM),asillustratedintheequationsbelow.
𝑀𝑆𝐸'T'2U =𝑌#5W 𝑥, 𝑦 − 𝑌'X' 𝑥, 𝑦
YK,L6,F
𝑁𝑀
𝑀𝑆𝐸'T'2U = 𝑀𝑆𝐸"#$%&' + 𝑀𝑆𝐸12#3 + 𝑀𝑆𝐸'56'7#5
𝑀𝑆𝐸"#$%&' =𝑊"#$%&' 𝑥, 𝑦 𝑌#5W 𝑥, 𝑦 − 𝑌'X' 𝑥, 𝑦
YK,L6,F
𝑁𝑀
Thevaluesof𝑀𝑆𝐸12#3and𝑀𝑆𝐸'56'7#5 maybecalculatedinasimilarmanner.TheresultingweightedMSEvaluesprovideinsightintotheproportionofthetotalMSEmaybeattributedtobright&darkfeaturesandtextures.ThesamemethodologymaybeappliedtobothlumaandchromaMSEswithappropriatescalingforthe4:2:2YCbCrformat.
WeightedMSEresultsfortheHDRWCGtestsequencesthatareplottedinFigure16demonstratethatthemajorityofthetotalMSEisattributabletothetexturecomponent.WefoundthisconclusiontobeconsistentacrossallHDRWCGtestsequencesforallcompressionlevelswetestedandthattheconclusionholdsforlumaandchroma.ThedominanceoftextureMSEismainlyaresultoftexturemakingupthelargestproportionofvideoframes(seeFigure14).
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 18
Figure16-WeightedMSEforBright&DarkFeaturesandTextures
Squared-ErrorDensityIntroductionofaSquared-ErrorDensity(SED)providesameansofselectivelyprobingdistortionforfeaturesandtextureswhileaccountingforeachone’srelativeprominenceinHDRWCGvideo.SEDmaybecalculatedforbright&darkfeatures,andtexturesaccordingtothefollowingequations:
𝑆𝐸𝐷"#$%&' =L\]@ABCDE^@ABCDE
;𝑆𝐸𝐷12#3 =L\]MNAO^MNAO
;𝑆𝐸𝐷'56'7#5 =L\]EPIEQAP
EPIEQAP
SEDisMSEdividedbythecorrespondingproportionalityoffeatureortexture.SEDthusaccountsforthefactthatfeaturestendtoberarerthetexture(seeFigure14).SEDmaybethoughtofasprovidingameasureofequitabilitybetweenfeaturesandtextures.Forexample,SEDcanprovideinsightintowhetherrarerfeaturesexperiencedisproportionatedistortioncomparedtotexture.
SEDresultsfortheHDRWCGtestsequencesareplottedinFigure17.Wefindsquared-errordensityforbrightanddarkfeaturesisrelativelymoreseverethanfortextures.ThisfindingisconsistentforallHDRWCGtestsequencesandcompressionlevelswetestedandholdsforlumaandchroma.
Figure17-Squared-ErrorDensityforBright&DarkFeaturesandTextures
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 19
CONCLUSIONWehavepresentedinthispaperasetofvideodistortionsmetricsthatmightprovetobeparticularlyusefulforHDRWCGvideo.Themainmotivatingprinciplewepresentedwasthe“SpatialDetail”signalthatweusedintwoways:1)asaproxyfortheoriginalimagedata;and2)asaguidetotheperceptuallyimportant“features”and“textures”inHDRWCGvideo.
TheSpatialDetailsignalisacondensedversionoftheoriginalimagethatpreservestherecognizabledetailsinanimagewhilediscountinglocalluminance.Itcanbethoughtofasatrue2-dimensionaldifferentialoftheoriginalimage.Itmayalsobeunderstoodintermsofthephaseinformationinanimageinconjunctionwiththestatisticallyunpredictableinformationinanimage.Fromapracticalstandpoint,itdoesn’treallymatterwhichtheoryoneprefers.Instead,animportantkeycharacteristicoftheSpatialDetailsignalisthatithasastatisticallystableandexpectablecompact,peaked,biphasic,andsymmetricdistributionofvaluesthatispreservedacrossawiderangeinimagesandvideo.Largervalues–positiveandnegative–formaconvenientguidetothekindsoffeaturespeopletendtofindsignificant.SpatialDetailvaluesnearerthezeromidpointofthedistributionformaconvenientguidetoimageregionsthatpeoplewouldtendtoclassifyastextural.SuchfeatureandtexturemapsprovideastableframeworkinwhichtoselectivelyinvestigatetheperceptualpotenthighlightsanddarkdetailsthatarethehallmarkofHDRWCGvideo.
WepresentedthreeHDRWCGvideodistortionmetricsinthispaper:
1. Forthefirstmetric,weusedSpatialDetailasaproxyfortheoriginalimageandshowedthatcorrelationbetweentheSpatialDetailsignalsoftheuncompressedandcompressedversionsofHDRWCGvideowassystematicallyaffectedbytheaggressivenessofHEVCcompression.BycombiningSpatialDetailcorrelationwithourfeatureandtextureassignmentmethods,weshowedthattexturecorrelationwasimpactedsignificantlymorethanfeaturecorrelation.SpatialDetailcorrelationhasseveraldistinctionswhencomparedtoestablishedvideoqualitymetrics.Itcanbeusedselectivelyonbright&darkfeaturesandontextures.Moreover,SpatialDetailvaluesareintherangeof0to1,whichismoreintuitivethantheunboundedPSNRscale,whilebeingamuchmoresensitiveindicatorthanMS-SSIMovertherangeofcompressionlevelstypicalofcableandpayTVoperations.
2. Forthesecondmetric,weusedSpatialDetailasaguideforbright&darkfeaturesandtexturetoselectivelyquantifytheMSEforeachlayerofimagedetail.WeshowedthattextureisthelargestcontributortooverallMSEmainly,becausetextureregionstypicallymakeupalargerproportionofanyimagethantherarerfeatureregions.
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 20
3. ThethirdmetricintroducedaSquared-ErrorDensity(SED)thatcompensatesfortherelativeproportionsoffeatureandtextureinanimagesoasassessdistortionsonamoreequalscale.WefoundthatSEDindicatesthatfeaturesexperiencedisproportionatedistortioncomparedtotexture.
Wehavedeliberatelyusedtheterm“videodistortion”insteadof“videoquality”throughoutthispaper.Themainreasonfordoingsoisthatthemetricsweproposedhavenotyetbeencomparedtosubjectivetestscoresandthusmaynotyetbeclaimedtobecalibratedsubjectivequalitymetrics.Also,itisnottheintentofthispapertolinkthemetricsweproposetosubjectiveassessment;thoughwemaydosoinlatterpublications.Rather,ourintentistoprovideeasytocalculatemetricsthatwehopecanprovideinsightduringthiscriticalperiodinourindustryasweworkthroughthetechnicalandcreativeissuesrelatedtoHDRandWCG.
ItisalsoworthhighlightingthattheSpatialDetailsignalandrelatedmetricsareeasytocalculateusingmodernsignalprocessingtechniquesinmodernprocessors.Thus,webelievethetechnicalbarriertoadoptionofthesemetricsislow.
Ourintentinthepaperistoprovideusefulandeasy-to-calculatemetricsthathavealowtechnicalbarriertoadoption.TheSpatialDetailsignalandrelatedmetricsweproposeareeasyenoughtocalculatethattheyarecandidatesforreal-timeHDRWCGvideoassessmentusingmodernsignalprocessingtechniquesinmodernprocessors.OurnextstepswillbetocontinuetoassesstheutilityofourHDRWCGmetricswiththehopethattheywillhelpMSOsnavigatekeytechnicalandcreativeissuesasHDRWCGvideoprogrammingemergesasthenextwaveofgreatsubscriberexperiences.
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 21
ABBREVIATIONSFFT2 2-dimensionalFastFourierTransformFSIM Feature-SimilarityIndexHDR HighDynamicRangeHEVC HighEfficiencyVideoCodingICTCP ICTCPcolorspaceIFFT2 Inverse2-dimensionFastFourierTransformMSE MeanSquareErrorMSO MultipleSystemsOperatorsMS-SSIM MultiscaleStructuralSimilarityPQ PerceptualQuantizerPSNR PeakSignal-to-NoiseRatioPU PerceptuallyUniformSDR StandardDynamicRangeSED Squared-ErrorDensitySSIM StructuralSimilarityYCbCr YCbCrcolorspaceVDP VisualDifferencePredictorVIF VisualInformationFidelityVQM VideoQualityMeasureYUV YUVcolorspaceWCG WideColorGamut
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 22
RELATEDREADINGS• ASystematicApproachtoVideoQualityAssessmentandBitratePlanning–In
thispaper,theauthorpresentsastreamlinedmethodofsettingoperationalvideoqualityandbandwidthusingeithersubjectiveorobjectivetesting,usingindividualgolden-eyesorfocusgroupsofanysize.ThedataandanalysisincludedareintendedtoaidinplanningvideoqualityandbandwidthresourcesacrossarangeofserviceofferingsfromOTTthroughUltraHD.
• EfficientContentProcessingforAdaptiveVideoDelivery–Thispaperprovidesanin-depthoverviewoftwoemergingtechnologies,dynamicprofileselectionandcooperativetranscoding,alongwithexperimentaldatademonstratingtheirpotentialforsubstantiallyreducingcontentprocessingrequirementsformultiscreenvideodelivery.
• MethodologiesforQoEMonitoringofIPVideoServices–ThispaperdiscussesthedifferencesbetweenQoEandQoSandbetweenQoEandvideoqualityandthencomparesdifferentmethodologiesforvideoqualityandQoEmonitoring.ItalsoincludesareviewofalternativesforembeddingQoEprobesintheend-to-endIPVideoarchitectureandtheirabilitytocollecttrueandeffectiveQoEinformation.
MEETOUREXPERT:SeanT.McCarthyDr.SeanMcCarthy,FellowoftheTechnicalStaff,bringsauniqueconvergenceofexpertiseinvideocompression,signalprocessing,andtheneurobiologyofhumanvisiontocontentdistributionatARRIS.Dr.McCarthyleadsadvancementsinstate-of-the-artofvideoprocessing,compressionandpracticalvisionscience.Previously,heheldsimilarresponsibilitiesasFellowoftheTechnicalStaffatMotorola,andasChiefScientistatbothModulusVideo,whichwasacquiredbyMotorola.Priortothat,Dr.McCarthyhadsimilarresponsibilitiesatViaSense,aUniversityofCalifornia,Berkeleyspin-offthatdevelopedcommercialapplicationsofthehumanvisualsystem.HeearnedaB.S.inphysicsfromRensselaerPolytechnic,andearnedhisPh.D.inbioengineeringjointlyatUniversityofCalifornia,BerkeleyandUniversityofCalifornia,SanFrancisco.
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 23
REFERENCES(1) Hanhart,P.,Kroshunov,P.,andEbrahimi,T.“Subjectiveevaluationofhigherdynamicrangevideo.”ProceedingsofSPIE-TheInternationalSocietyforOpticalEngineering,2014
(2) Wang,Z.,Bovik,A.C.,Sheikh,H.R.,andSimoncelli,E.P.“Imagequalityassessment:fromerrorvisibilitytostructuralsimilarity,”IEEETransactionsonImageProcessing,vol.13,no.4,pp.600–612,Apr.2004
(3) Hanhart,P.,Bernardo,M.V.,Korshunov,P.,andPereira,M.“HDRimagecompression:Anewchallengeforobjectivequalitymetrics.”inSixthInternationalWorkshoponQualityofMultimediaExperience(QoMEX),2014
(4) ITU-RBT.500-13,“Methodologyforthesubjectiveassessmentofthequalityoftelevisionpictures,”InternationalTelecommunicationUnion,Jan.2012
(5) ITU-TP.910,“Subjectivevideoqualityassessmentmethodsformultimediaapplications,”InternationalTelecommunicationUnion,April2008
(6) Winkler,S.DigitalVideoQuality:VisionModelsandMetrics,JohnWiley&Sons,Mar.2005
(7) VQEG,“Finalreportfromthevideoqualityexpertsgrouponthevalidationofobjectivemodelsofvideoqualityassessment,”Mar.2000.http://www.vqeg.org/.
(8) Wang,Z.andBovik,A.C.“Meansquarederror:loveitorleaveit?-Anewlookatsignalfidelitymeasures,”IEEESignalProcessingMagazine,vol.26,no.1,pp.98-117,Jan.2009
(9) Hanhart,P.,Bernado,M.V,Pereira,M.,Pinheiro,M.G.,andEbrahimi,T.“BenchmarkingofobjectivequalitymetricsforHDRimagequalityassessment.”EURASIPJ.Image&VideoProcessing,2015
(10) Wang,Z.,Simoncelli,E.P.,andBovik,A.“Multi-scalestructuralsimilarityforimagequalityassessment.”Proc.ofthe37thIEEEAsilomarConferenceonSignals,Systems,andComputers,2003
(11) Sheikh,H.R.andBovik,A.C.“Imageinformationandvisualquality,”IEEETransactionsonImageProcessing,vol.15,no.2,pp.430–444,Feb.2006
(12) Field,D.J.“Relationshipbetweenthestatisticsofnaturalimagesandtheresponsepropertiesofcorticalcells.”J.Opt.Soc.Am.A.Vol.4,No.12,1987
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 24
(13) Nill,N.B.andBouzas,B.H.“Objectiveimagequalitymeasurederivedfromdigitalimagepowerspectra.”OpticalEngineering,vol31,no.4,1992
(14) Liu,r.andLaganiere,R.“Ontheuseofphasecongruencytoevaluateimagesimilarity.”IEEEInternationalConferenceonAcousticsSpeechandSignalProcessingProceedings,2006
(15) Liu,r.andLaganiere,R.“Phasecongruencemeasurementforimagesimilarityassessment.”PatternRecognitionLetters,vol28,no.1,2007
(16) Kovesi,P.“ImageFeaturesfromPhaseCongruency.”inVidere:JournalofComputerVisionResearch,Vol1,No.3,TheMITPress,1999
(17) Kovesi,P.“Invariantmeasuresofimagefeaturesfromphaseinformation.”Thesis(Ph.D.)Dept.ofComputerScience.UniversityofWesternAustralia,1996
(18) Morrone,M.C.andOwens,R.A.“Featuredetectionfromlocalenergy.”PatternRecognition.”Lett.,303–313,1987
(19) Morrone,M.C.andBurr,D.C.“Featuredetectioninhumanvision:Aphase-dependentenergymodel.”Proc.RoyalSoc.OfLondon,SeriesB,BiologicalSciences,vol.235,no.1280,1988
(20) Zhang,L.,Zhang,L.,Mou,X.,andZhang,D.“FSIM:Afeaturesimilarityindexforimagequalityassessment.”IEEETrans.ImageProcess.vol20,no.8,2011
(21) Mantiuk,R.,Daly,S.,Myszkowski,K.,andSeidel,H.-P.“Predictingvisibledifferencesinhighdynamicrangeimages:modelanditscalibration.”SPIEHumanVisionandElectronicImagingX,vol.5666.,2005
(22) Daly,S.J.“Visibledifferencespredictor:analgorithmfortheassessmentofimagefidelity”SPIEHumanVision,VisualProcessing,andDigitalDisplayIII,vol.1666.,1992
(23) Mantiuk,R.,Kim,K.J.,Rempel,A.G.,andHeidrich,W.“HDR-VDP-2:Acalibratedvisualmetricforvisibilityandqualitypredictionsinallluminanceconditions.”ACMTrans.Graph.30(4),40:1–40:14,2011
(24) Narwaria,M.,Mantiuk,R.K.,PerreiraDaSilva,M.,andLeCallet,P.“HDR-VDP-2.2:acalibratedmethodforobjectivequalitypredictionofhigh-dynamicrangeandstandardimages.”J.Electron.Imaging.24(1),010501,2015
(25) Aydin,T.O.,Mantiuk,R.,andSeidelH.-P.“Extendingqualitymetricstofullluminancerangeimages.”HumanVisionandElectronicImagingXIII.EditedbyRogowitz,BerniceE.;Pappas,ThrasyvoulosN.ProceedingsoftheSPIE,Volume6806,2008
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 25
(26) Miller,S.,Nezamabadi,M.,andDaly,S.,“PerceptualSignalCodingforMoreEfficientUsageofBitCodes.”SMPTEMotionImagingJournal,2013
(27) Valenzise,G.,DeSimone,F.,Lauga,P.,andDufaux,F.“PerformanceevaluationofobjectivequalitymetricsforHDRimagecompression.”Proc.SPIE9217,ApplicationsofDigitalImageProcessingXXXVII,2014
(28) Mantel,C.,Ferchiu,S.C.,Forchhammer,S.“ComparingsubjectiveandobjectivequalityassessmentofHDRimagescompressedwithJPEG-XT.”in16thInternationalWorkshoponMultimediaSignalProcessing(MMSP),IEEE,2014
(29) Rerabek,M.,Hanhart,P.,Korshunov,P.,andEbrahimi,T.“SubjectiveandobjectiveevaluationofHDRcompression.”InternationalWorkshoponVideoProcessingandQualityMetricsforConsumerElectronics-VPQM,Chandler,Arizona,USA.February2015
(30) McCarthy,S.T.andOwen,W.G.“ApparatusandMethodsforImageandSignalProcessing”.USPat.6014468(2000).USPat.6360021(2002),USPat.7046852(2006),1998
(31) McCarthy,S.,“ABiologicalFrameworkforPerceptualVideoProcessingandCompression,”SMPTEMot.Imag.J.,119(8):24-32,Nov/Dec.2012
(32) McCarthy,S.T.“Theoryandpracticeofperceptualvideoprocessinginbroadcastencodersforcable,IPTV,satellite,andinternetdistribution.”Proc.SPIE9014,HumanVisionandElectronicImagingXIX,2014
(33) Froehlich,J.,etal.“HdM-HDR-2014Project,”http://www.hdm-stuttgart.de/~froehlichj/hdm-hdr-2014
(34) Froehlich,J.,Grandinetti,S.,Eberhardt,B.,Walter,S.,Schillin,A.,andBrendel,H.“CreatingcinematicwidegamutHDR-videofortheevaluationoftonemappingoperatorsandHDR-displays.”Proc.SPIE9023,DigitalPhotographyX,2014
(35) ITU-RBT.2020“Parametervaluesforultra-highdefinitiontelevisionsystemsforproductionandinternationalprogrammeexchange."InternationalTelecommunicationUnion,2012
(36) TheMathworks.www.mathworks.com
(37) ffmpeg.www.ffmpeg.org
(38) x265.www.x265.org
(39) ITU-RBT.2100.“Imageparametervaluesforhighdynamicrangetelevisionforuseinproductionandinternationalprogrammeexchange.”2016
Copyright2016–ARRISEnterprisesLLC.Allrightsreserved. 26
(40) ReportITU-RBT.2390-0“Highdynamicrangetelevisionproductionandinternationalprogrammeexchange.”InternationalTelecommunicationUnion,2016