Open Research OnlineThe Open University’s repository of research publicationsand other research outputs

Demosaicing images from colour cameras for digitalimage correlationJournal ItemHow to cite:

Forsey, A. and Gungor, S. (2016). Demosaicing images from colour cameras for digital image correlation.Optics and lasers in engineering, 86 pp. 20–28.

For guidance on citations see FAQs.

c© 2016 Elsevier

Version: Accepted Manuscript

Link(s) to article on publisher’s website:http://dx.doi.org/doi:10.1016/j.optlaseng.2016.05.006

Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.

oro.open.ac.uk

Demosaicingimagesfromcolourcamerasfordigitalimagecorrelation

A.Forsey*,S.Gungor

TheOpenUniversity,MiltonKeynes,Bucks,UK,MK76AA

AbstractDigital ImageCorrelationisnotthe intendeduseforconsumercolourcameras,butwithcaretheycanbesuccessfullyemployedinsucharole.Themainobstacleis the sparsely sampled colour data caused by the use of a colour filter array(CFA) to separate the colour channels. It is shown that the method used toconvertconsumercamerarawfilesintoamonochromeimagesuitablefordigitalimagecorrelation(DIC)canhaveasignificanteffectontheDICoutput.Anumberofwidelyavailablesoftwarepackagesandtwoin-housemethodsareevaluatedin terms of their performancewhenusedwithDIC. Using an in-plane rotatingdisc to produce a highly constrained displacement field, it was found that thebicubic spline based in-house demosaicing method outperformed the othermethodsintermsofaccuracyandaliasingsuppression.Keywords:DigitalImageCorrelation;Demosaicing;ColourFilterArray;ColourCamera.

1 IntroductionDigital Image Correlation (DIC) [1,2] is an increasingly popular technique formeasuring spatially resolved surface strain. Its principle is based oncomputationaltrackingofcontrastingsurfacefeaturesondigitalimages.Oneofthe main attractions is the relatively simple equipment required and that issimplyadevicecapableoftakingsuitableimages,mostoftenacamera.Typically,two or more images are acquired before and after a loading event and therelativemovementsof thesurface features ineach imagearedetermined.Highmeasurement accuracy relies on resolving features at a sub-pixel level. This isbestachievedwhenthelightintensityofeachpixelisregisteredaccurately,asinthe caseofmonochromecameras. It ispossible touse colour cameras forDIC,but first the colour informationmust be converted into amonochrome signaland themethod bywhich this is achieved can have a significant effect on theeventual result. Therefore, scientific monochrome cameras are predominantlyusedforDICsothatthisprocessingstepcanberemovedandbecauseonapixel-to-pixelcomparisonthecolourcamerasareatadisadvantage.TherehasbeenalargebodyofworkdedicatedtothemeasurementorestimationoferrorinDIC.This work generally considers the error caused by different algorithms [3],differentpartsofthealgorithm[4,5],ormethodsforestimatingerror[6].Totheauthors’knowledgetherearenopublishedstudiesontheeffectsofusingcolourcameraforDICotherthanYoneyama[7],whousea3CCDcolourcamerarather

thanonewithacolourfilterarray(CFA).Thisarrangementremovestheproblemof sparse sampling of each colour channel, but does not represent theconfigurationofthemajorityofdigitalcolourcameras.ForcamerasusingCFAs,only statements in papers that have used colour cameras that allude to theireffects[8]havebeenfound.Therearesituationswhereusingacolourcameracouldhaveanadvantage,butthis is mainly down to grounds of cost. Due to the large market demand forcolourcameras,highqualitymodelscanbeobtainedforafractionofthecostofdedicatedscientificcameras.Theseconsumercamerashaveafasterproductlifecycleandcanpossessalargepixelcount.Thesecamerasarenotdesignedtotakescientific measurements and so they have multiple undesirable features notfoundonscientificcameras.Thesearespecifically,andnotlimitedto,CFAs,pixellenses, anti-aliasing filters, and a viewfinder mirror mechanism. However, forlong term testing where a camera is in place for months [9,10], the cheapercolour cameras may still be an attractive proposition. With the additionalresolution of these cameras, there are other situations in which they maypreferabletothemonochromescientificcameras,suchascrackdetection.Therealsomustbeacomparableresolutionatwhichagoodqualitycolourcamerawillgainperformanceparitywithamonochromecameraofa lowerresolution,dueto the higher resolution colour camera being able to use more pixels is eachsubregiontoobtainthesamespatialresolution.

2 DemosaicingWhenperformingDIC,amonochromecameraispreferableoveracolourcameraofthesameresolution,unlessthecolourinformationisrequiredforaseparatepurpose. These cameras are made up of photosites that all have similarsensitivityandsoanyspecklemovingfromonepixeltothenextwillproduceasimilarandpredictable response.Themajorityof colour camerasuseaCFA tomakeindividualpixelssensitivetored,greenorblue(somecamerasseparatetoCMYK,buttheprincipleremainsthesame).Thearrangementof theCFAof thecolourcameradescribed in thispaper isaBayerpattern[11].Thispatternhastwice the number of green photosites as red or blue arranged in a 2x2pixelrepeating-unit, as seen in Figure 1. From this sparse colour sampling, a fullcolour image is produced by interpolating the unknown values in each colourchannel. This interpolation is achieved via any one of the many demosaicingalgorithmsavailable [12–14] to calculatea red, greenandbluevalue foreverypixel position, where only data from one channel was captured. These threechannels, red, green and blue, can then be combined to create amonochromeimagesuitableforDIC.TherequirementsforthedemosaicingprocesstoperformsuccessfulDICusingacolour camera are somewhat different to that of that to create a successfulphotograph. Amonochromeoutputfromthecameraisrequiredandtheresultshould be as repeatable as possiblewhen the sample is subjected to sub-pixelshifts. The first condition is simple to achieve, the second is much moreproblematicduetothesparsesamplingofeachcolourchannel.Oneoftheaims

ofthisstudyistoinvestigatetheeffectofhowthecolourchannelsarecombinedtoproduceamonochromeimageforDICandtheresultingoutput.For consumer cameras, the rawmosaic image can be accessed though savingimages in the proprietary raw format. Many algorithms are available forproducing a full colour image from these sparsely sampled colour channelscausedbytheCFA.ForDIC,amonochromeimageisrequiredasthecorrelationisperformedonasingleregulararrayofdata.Thisissothatthespecklepattern,specificallythedifferenceinlightintensitybetweencontrastingfeatures,movesfromonepixeltothenextasdisplacementincreases.TheaimofthedemosaicingconversionistoallowaDICalgorithmtomakethebestuseoftheavailabledatafrom the three colour channels. Two in-house methods will be tested here,bilinear interpolation and bicubic spline interpolation. These two are thenbenchmarked against a typical commercial package, in this case Corel PhotoPaint [15].Fiveotheralgorithms from theopensourceRAWtherapee softwarearethenconsideredandallalgorithmsusedaresummarisedinTable1.Theaimof this is not to promote or condemn the Corel software or an open sourceapproach, merely to provide context for the other methods presented. CorelPhotoPaintwaschosenbecauseofitsabilitytoproduceasuitabletiffilethatcanbe read by the LaVision software (16bit,monochrome, uncompressed tif) in asingle program and as such would be a convenient choice for any user. Theconversion using Corel Photo Paint was performed using software’s defaultsettings of sharpness and colour balance. RAWtherapeewas chosen due to itsrange of demosaicing algorithms from a single open source. In this case theimageswereconvertedtomonochrometifsusingMatlab,givingequalweighttoeachcolourchannel.Demosaicing method Comment Ref

Ahd Adaptive homogeneity directed [16] AMaZE AliasingMinimizationandZipperElimination [17] Bicubic Bicubic spline interpolation in-house

Corel PP X4 Proprietary software [15] dcb Góźdźmethod [18] eahd Horváth’sAHD [19]

Linear Bilinear polynomial interpolation in-house vng4 VariableNumberofGradients [20]

Table1:Summaryofdemosaicingalgorithmsconsideredinthisinvestigation.

With theaimtoproduceaspatiallyconsistentmonochrome image, the two in-housemethods considered here treat each colour channel entirely separately.Theinterpolationisperformedintwoways,thefirstusingbilinearinterpolationandthesecondusingbicubicsplines.Forbothmethods,eachmissingcomponentofthecolourchannel iscalculatedusingadifferentsizeofregionforthegreenchannelas for theredorblue.For thebicubicsplinemethod, theextentof theregionsrequiredfor interpolationof thedifferentcolourchannels is illustratedschematicallyinFigure2.Thisfigurealsoillustratesthattocalculatethesizesoftheregionsrequiredtocalculatetheredandbluechannelsatagreenphotositehave rotational symmetry of order 2, rather than 4 as for the other twosituations. The splines use onlymeasured values in a single colour channel as

knots and are defined using the not-a-knot end condition. This process pathproducesagreenchannelthatpossessestwicethespatialfrequencyinformationof either the red or the blue channels. Once the RGB channels have beencalculated, the channels are summed giving double weighting to the greenchannelduetoithavingtwicethenumberofinputmeasurementpoints.As the CFA is not easily removed to provide a direct comparison of theperformance of a truemonochrome camera, a second camera will be used toperform this role.By comparing these twocamerasperhapsamore importantcomparisoncanbemadebetweenthetwopotentialoptionsfacedbyDICuserswhenselectingcamerasfortheirneeds.

3 ExperimentalTo enable a direct comparison between different demosaicing algorithms andcameras,atest isrequiredwithawell-characterisedsolution.Forthisreasonaplatewithapaintedspecklepatternappliedtothesurfacewasrotatedaboutthenormal of the lens in the object plane. Once the displacement vectors wereproducedusingDIC,a first-ordertwo-dimensionalpolynomialwasfittedtothedata using the least-squaresmethod.No deformationwas applied to the plateand it was rotated in plane and so, due to the large number of nearlyindependentdatapointsacrosstheimage,thefittedplanecanbetakenasaveryclose approximation to the true displacement of the plate. The calculateddisplacementvectorscanthenbesubtractedfromthe“true”displacementoftheplatewiththeresulthenceforthreferredtoasleast-squareserror(LSE).

3.1 CamerasBoth the cameras tested use the same lens-mount and have a similar sizedsensor so a direct comparison is possible with the same optical setup. Thisresultsinaverysimilarregionofinterestavailableforeachforeachcamera.Thepixelcountandthereforepixelssizewilldiffer,butthecamerasarethehighestresolution colour and monochrome cameras currently available with thismounting system. For this reason, comparing a 16-megapixel monochromeimagetoa36-megapixelcolourimageisreasonableastheaimistocomparethemaximum achievable DIC resolution. The drawback of this method forcomparison is that thetwocameraswillproduce imageswithdifferentspecklesizes in terms of pixels. Tomitigate this the in-house demosaicing algorithmswere also used on themonochrome images by treating the image as if it hadbeentakenwithaCFAonthesensor.ThecamerasusedinthisinvestigationwereallmanufacturedbyNikon,thiswasbecause they use the same “F-mount” and so could be interchanged withoutchangingthefocusorworkingdistanceofthelens.Asimilareffectmaybeseenusingcamerasmadebydifferentmanufacturers,butthathasnotbeenaddressedhere.TheNikonDSQi2 isa16-megapixelmonochromemicroscopecamerabasedonNikon’s consumer camera technology and CMOS sensors. However, it differsfrom the DSLR (Digital Single Lens Reflex) cameras in that the sensor also

resides in a number ofways thatmake itmore suitable forDIC. Primarily theCFA is removed so it is a truemonochrome camera; similarly the anti aliasingfilterandpixelmicro-lenseshavealsobeenremoved.Thebody isdesigned foruseonamicroscopeandsonomirrororphysicalshutterarepresenttoinducevibration into the system.The sensor is alsoelectronically cooled toprevent abuiltupofheatdue toprolongeduse from increasing the readoutnoise in theimage.TheNikonD810digitalsinglelensreflex(DSLR)cameraisa36megapixelcolourconsumercamera.IthasaCFAusingaBayerpatternwithindividualmicrolensesoneachphotositetoincreasetheamountoflightgatheredforeachpixel.Thesemicrolenseshelpincreasetheeffectivefillfactorofthesensorattheexpenseofuniformityofresponseacrossthelightsensitiveregionofthephotosite.Thesameopticalpathsetupwasusedforalltests.ThiswascomprisedofaNikon200mmmicrof4lens.Illuminationwasprovidedusinga150Wlightboxwitha0.5” diameter fibre optic light guide so no heat or thermal currents wereintroduced into the imaging system and a working distance of 600mm. Thespecklepatternwasproducedona1mmthickaluminiumsheetusingmattanti-rustaerosolpaint.Firstawhitebasewasappliedacrossthesurface,followedbyablackspeckle.Manypassesof theblackpaintwereused toslowlybuildupadense speckle pattern. The paint was not sprayed directly toward the samplesurface,insteaditwasaimedbelowsothatonlythesmalllightpaintparticlesintheoverspray formpartof thespecklepattern. In thiswaythespecklepatterncanbekeptmoreconsistent.

3.2 ProcedureTheexperimentalprocedureinvolvedincrementallyrotatingthespecklepatternusing a rotation stage and taking images at each rotation step. In order toprevent bias in the error analysis to be introduced by small differences inrotationitwasdecidedtoremoveandreplacethecamerasateachrotationstep.Withthelensfirmlymountedtothesamerailastherotationstagethecameras(see Figure 3) could be replaced more accurately than the rotation could bereproduced. The cameras were removed and replaced before each imagewastakensoeachimageunderwentthesamedisturbance.Sixteenimagesforeachcamerawererecordedandpassedthroughthevariousdemosaicing filters to produce 11 sets of images. Each set of images wasprocessedusingDaVis8.2[21]leastsquaresmethod[2,22,23]usingparametersoutlinedinTable2,forsubregionsizesof11,15,21,31,41,51and61,allata7pixelstepsize.ThesedatawerethenexportedtoMatlab[24]whereaplanewasfittedtothedata.

Algorithm parameter Value Pyramid levels 1

Epsilon 0.01 Correlation threshold 0.2

Subregion shape Round Subpixel interpolation Bicubicspline

Subregion scale normalisation On Step size 7

Table2:SummaryofcommonDICalgorithmparametersforallanalyses.

Toprovidecontexttothecomparisonbetweenthemonochromecameraandthedemosaicedcolourimages,twoofthedemosaicingalgorithmshavebeenappliedto the monochrome images. The monochrome image was split into threechannels, each one an analogue of a colour channel from a colour camera andtakenfromthesamepositionontheimagesensor.Thisproducedtwochannelswithsimilarnumbersofdatapointstorepresenttheredandbluechannelsandathird channel with twice as many data points to represent the green. Thesemosaicedimageswerethenprocessedusingthelinearandbicubicdemosaicingalgorithms used on the true colour cameras. In this way the demosaicingalgorithms can be directly compared with monochrome data from the sameimage.

4 ResultsThe global measured displacement as calculated for each camera anddemosaicing algorithm using the polynomial fit, can be seen in Figure 4a. Theincreaseinrotationisnotlinearandappearstohavetworotationstepsthataresignificantlysmallerthantherest,specificallyframes2to3and10to11.Overallvalues of rotation are extremely consistent for each camera across differingdemosaicing algorithms. There is a small discrepancy between the results ofrotationfromthemonochromecameraandthecolourcamera.Figure4bshowsthe absolute difference in the measure of rotation between the differentapproaches is at a maximum for smaller values of rotation and decreases asmorerotationisapplied.Thevaluesinthisplotareallrelativetotheaverageforthe appropriate image (the value from Figure 4a) and so should be taken ascomparativeasthe“true”valueisnotknown.Measurement of the in plane rotation of a flat plate produces a highlyconstrainedvectorfield.Thiswasdonetosimplifytheerroranalysis,butitalsohasthebenefitthattheerror,andtheprogressionoftheerror,canbevisualisedby considering the asymmetric parts of the strain tensor. Figure 5 shows theprocessionof error (Least SquareError –LSE) across the surface fordifferentapplied rotation for the 61x61 pixel subregion analysis of the y-directiondisplacementsofthecolourcameraimages.Aliasingcanbeseenintheformofthe repeating peak-valley “dotted” pattern. For each algorithm, this patternremainssimilarinmagnitudebutincreasesfrequencyasrotationisapplied.Itisalsoapparentthatthedemosaicingmethodhasalargeeffectontheamplitudeofaliasing observed, with the two in-house algorithms clearly producing lessprominentaliasingeffectsthantheothermethodsconsidered.Fromthesamplespeckle shown for each analysis it can be seen that the monochrome cameraproducesahighercontrastimagethantheprocessedcolourcameras.Figure 6 shows the common aliasing, or peak-locking, effect [25] as seen togreater or lesser extent for all cameras used for DIC, caused primarily byinterpolationerror[5].Thisplotshowsthehistogramofthenormalisedpercent

ofvectorsfromall15imagesthatarepresentinthedataattwentyincrementalsub-pixeldisplacementvalues.Thisshowsthedifferenceinthemeasuredtotheexpectednumberofvectorspresentineachbin.Partaoftheplotshowsvaryingdegrees of integer pixel bias for the 11x11 pixel subregions for the differentcameras and demosaicing algorithms. The generic algorithms suffer from agreateramountofaliasingthanthein-houselinearandbicubicalgorithms.Therawmonochromecamerahas thehighestoverall degreeof aliasing and this isreducedbytheapplicationofthein-housedemosaicingalgorithms.Partbofthisfigureshowsthatthiseffectnolongerpresentusingthe61x61pixelsubregions.Figure7showssimilardatatoFigure6,butinsteadofbinningoverasinglepixelall thedata isbinned in20 incrementsacross2pixels.This isdonebecauseofthe2x2repeatingminimumunitoftheCFAonthecolourcamera.Itisclearfrombothpartsaandbofthefigurethatthecolourcamerasuffersconsiderablymorealiasing at this scale than the monochromatic comparison. The genericdemosaicingalgorithmsshowgreaterbi-integerbias than the linearorbicubicin-house algorithms,with thedcbmethod showing the greatest control of thiseffect. Themonochrome camera, aswould be expected due to its lack of CFA,showsnobi-integerbiaseffect.Inthisfigureitrevealsitsintegerbiasasaneffectattwicethefrequencyofthecolourcameras.Theeffectisnotpresentevenwhenthemonochrome rawdata is put through the samedemosaicing algorithmsasthecolourcameradata.Bi-integerbiasisreducedforthe61x61pixelsubregionsinpartbofthefigureincomparisontothatinpartaat11x11pixels.Figure8athevariationofthezerodisplacementvalueswiththesubregionsize.These were computed from the histograms at each subregion size, of which11x11 and 61x61 are from Figure 6a and 4b, respectively. This shows therelative degree of aliasing present for different cameras and algorithms at thevarious subregion sizes. The integer bias rapidly drops to zero for thedemosaicedmonochrome images,while the rawmonochrome imagescontinueto reduce throughout the subregion size range. The generic algorithms showhigherlevelsofintegerbias,whichdoesnotchangeconsiderablywithsubregionsize.Thelinearalgorithmshowstheoppositetrendtoothermethodsinthatthepeakvalueincreaseswithincreasingsubregionsize.Thebicubicapproachshowsastablevalueofaliasingthatisbothlowerthananyoftheothercolourcameradataandalsomoreconsistent.Figure8b, similar toFigure8a, shows thezerodisplacementvalues for thebi-integer bias histograms of Figure 7a, Figure 7b, and those subregion sizesbetween.Themonochromecamera,bothrawanddemosaicedshowthe lowestlevels of bi-integer bias. Bicubic spline, linear and the dcb approaches showsimilarandconsistentlevelsofaliasingthatarevirtuallyinsensitivetosubregionsize. The remaining algorithms show a significantly larger effect that reduceswithincreasingsubregionsize. Figure 9a shows the average least squares error (LSE), the absolute meandifference between the plane fit and the displacement vectors, for varyingsubregionsizesand forallalgorithms.Allmethodsshowagradualdecrease inLSE with increasing subregion size. For subregion sizes below 31x31 pixels,

bicubic and bilinear colour camera demosaicing show lower LSE than themonochrome camera. At subregion sizes greater than this the monochromecameraproducesalowervalue.Figure9bshowsthesamedataasFigure9a,butconverted into true object plane scaling in mm. The smaller pixel size of thecolourcamerareducesalltheLSEvaluesofthecolourcameratowellbelowthatofthemonochromecamera.InthisplotthelowestvalueofLSEisfoundfromthecolourcamerausingbicubicsplinedemosaicingforallsubregionsizes.

5 DiscussionThe rotation measured for each image, as shown in Figure 4a, shows adiscrepancy between themonochrome and colour cameras. This is due to theexperimentalmethodwhereeachcamerawasremovedandreplacedbeforeeachimagewastaken.Todothis,thecamerasarerotatedinrelationtothelenstoahardstoponthemount,soanyinconsistencyintheamountofrotationbetweenthe twocameraswill largelybedue to float in theclipmechanismat thishardstop.Themeasurementof rotation appearsnot tobe affectedby the choiceofdemosaicingalgorithmand this isnot surprisingdue to thenumberofvectorsinvolvedinthiscalculation.Thisvalidatestheassumptionthatafitusedinthiswaycanbeusedasthe“true”displacementforthesubsequentanalyses.Thetwostepsthatexperienceasmallerrotationstepthantheothers,specifically2to3and10to11,areseenequallybybothcamerasandsothisisanartefactofslackinthetake-upinthegearsoftherotationstageusedfortheexperiment.Abeamsplitter could have been used to eliminate this problem caused by float in themechanism, but then the optical path would differ between cameras and anychromatic aberration caused by the beam splitter would disproportionatelyaffectthecolourcamera.Figure 4b shows that the effect on rotation measurement of the differentdemosaicingapproachesisatitshighestforsmallrotations.Thegreaterspreadof sub-pixel displacement values and the low signal to noise ratio caused bymeasuringsuchsmalldisplacementsemphasisingsystematicbiasesislikelythedominant cause of this phenomena.However, the only difference between theanalysesforeachcameraisthedemsaicingprocess,asthesameimagesareused,sothismustbetheultimaterootoftheeffect.It isoutside thescopeof thisexperiment toproduceuniversalerrorvalues fordifferent cameras or algorithms. The data presented here are for comparativepurposes only and only hold true for this speckle pattern and the processingmethodsused.However,itcanbeusedtoratetherelativeperformanceofeachalgorithm.Thisisnotatruetestofcameraperformanceduetothedifferenceineffective speckle sizebecauseof thedifferingpixel sizesbetween the cameras,butthisdoesnotpreventsomeconclusionsbeingmade.In Figure6, thedemosaicedmonochrome image suffers less of an integerbiaseffect than the rawdata.This ismost likelydue to the smoothingeffect of thedemosaicing methods and not some intrinsic property of the demosaicingprocess.Asimilareffectwouldmostlikelyhavebeenobservedifthelenswasde-focussedby a small amount to blur the sharp contrasts of the painted speckle

pattern. That the monochrome camera shows a larger effect than any of thecolouralgorithmsowesmoretothedifferenceineffectivespecklesizebetweenthe two cameras. To perform this test, a balance had to be struck whenprescribingthespecklesizeonthesamplesurfacethatwillenablebothcamerasto perform well despite their difference in resolution: this is a result of thatcompromise.Thelinearandbicubicdemosaicingmethodsshowtheleastintegerpixelbiasforthis set of images, but as can be seen in Figure 8a, the linear method is notconsistentforallsubregionsizes.Itisnotclearwhythiswouldbethecaseasthedemosaicingmethoditselfisverysimilartothebicubicsplinealgorithm,whichdoesnotdisplaythiseffect.Bi-integerbiasispresentindifferingamountsforalldemosaicingalgorithmsforthe colour camera as seen in Figure 7. In Figure 8b it can be seen that themagnitude of the bias reduces with increasing subregion size for the genericdemosaicalgorithms,butisconstantforboththelinearandbicubicapproaches.This is because the former have been developed for increasing sharpness andsupressingcolourfringing,asopposedtospatialconsistency.Bi-integerbias is notpresent for themonochrome camera inFigure7 and theperiodicitythatisvisibleisduetothesingleintegerbiasasdiscussedpreviouslythatismanifestasafeaturewithtwicethefrequencyofthebi-integerbiasfromthecolourcamera.Bi-integerbias isnotevenproduced fromthemonochromeimagewhen it is split into threesimulatedcolourchannels toapproximate thecolour camera and processed using the two in-house demosaicing algorithms.Thisshowsthatthisbi-integerbiasisnotaproductofthesparsesamplingandrecombinationprocessthattakesplaceinthecolourcamera,ratheritiscausedbysomeotheraspectofthecolourcamerathathasnotbeensimulatedhere.Thelikely responsible feature is the CFA itself and the difference in contrast thatoccurs when part of a speckle moves from one coloured pixel to a differentcolouredpixel.Evenwithatrueblackandwhitespecklepattern,thedifferencein response of the photosites to different colours would produce differentintensities from the same input. For the colour camera there is also thecomplication of microlenses on each photosite to increase its sensitivity, buttheseareunlikelytoproducethebi-integerbias,as theeffectwouldbesimilarforeachcolourchannel.Thiswouldsuggestthatanalgorithmthatwouldworkinasimilarmannertothebicubicmethodconsideredhere,butthatcouldcorrectforthedifferenceincontrastasseenbythefiltersofintheCFAmightreducethiseffectstillfurtherandproduceanevenbettertoolforusewithDIC.Thebicubicsplinebasedalgorithmshowsareduction in integerandbi-integeraliasingincomparisontotheothermethodstestedhere.Theeffectofwhichcanbe seen arbitrarily in Figure 5. This illustrates the 2D nature of the bi-integerbias effect and how itmaymanifest in regions of constant strain, such as thisidealisedsituationoftherotatingdisk.The higher contrast images from themonochrome camera are a result of thebroader frequency sensitivityof themonochromesensor.Whenusinga colour

sensor the illumination isalwaysacompromisebecausethephotositesarenotuniformly sensitive to all frequencies of light, in particular they are leastsensitive to the blue end of the spectrum. The photosites that provide theinformationtomakeupthischannelareonlysensitivetothebluespectrumandsothispartoftheimageproducespoorercontrastthanthegreen.Asimilar,butlesssignificanteffectalsooccurswith theredchannel. If the illuminationweretuned to allow equal illumination for each colour channel, the contrast in thecolourcameraimagescouldbeimprovedandsoleadtoanimprovementinDICperformanceasaresult.InFigure9aitappearsthatthebicubicandbilineardemosaicedcolourcamerasoutperformthemonochromecameraonapixeltopixelbasis.Thisismostlikelyduetothedifferenceinspecklesizebetweenthetwocameras,inparticularthespeckle in the monochrome image being small, in combination with sharpdelineationsbetweenlightanddarkareas.Thiscomparisonisalsonotasdirectas between the differently demosaiced colour images because it is calculatedfrom a different image taken by a different camera and so no such strongconclusionscanbemade.Whatcanbesaid is that theLSEof themonochromecamera in this test is comparable to the colour camera using the very bestdemosaicing algorithm. Considering that the monochrome camera imagespossess a compromised speckle pattern to enable the comparison, it is mostlikely that the pixel-to-pixel performance in comparison to the colour camerawouldbeslightlysuperiorinmoredirectlycomparableconditions.LSEispredominantlydeterminedbythedegreeofaliasingpresentintheresults,aconvolutionofbothintegerbiasandbi-integerbias,andassuchasimilartrendcanbeseenintheperformanceofthedemosaicingalgorithmsbythismeasure.The difference with the LSE measure is that it permits comparison of thecombinedresultofalltheseeffectswithanoutputintermsofdisplacement.Thedifference between the raw monochrome camera results and the demosaiceddata from these images shows what an effect having a speckle pattern withsmoothtransitionsfrompeaktopeakcanimproveDICresults.LSEismeasuredinpixelsandsothiscanbescaledintoobjectplanecoordinatesinmillimetres, the result can be seen in Figure 9b. This shows thatwhile thealiasing is large for some of the algorithms, it is not large enough to entirelyoffset the increased resolution boasted by the colour camera. To get the samespatial resolution with the same sized region of interest, the monochromecamerashowsconsiderablymoreLSEatallsubregionsizes.AsLSEisprimarilylinkedtoaliasing,itshowsthatwhilethealiasingofsomeofthesedemosaicingalgorithms issignificant,whenscaledto theobjectplane it isstill less thanthelower resolution monochrome camera. This is partially because the samefractionofapixelaliasinghasasmallereffectinthecolourcamera,becausethepixelissmaller.Alsowhencomparingthecamerasatthesamespatialresolution,the colour camera uses more pixels and so a larger subregion for the samecalculation.TheseeffectscombinedtogethershowproducealargenetdifferenceintheDICperformanceofthetwocameras.

To recreate the speckles from the sample surface with as little positional orintensity bias as possible is the aim of a demosaicing algorithm when usingcolourcameras forDIC.Ascanbeseen fromtheresultspresentedhere, this isnot easy to achieve. The in-house developed bilinear and bicubic algorithmsoutperformtheproprietaryandopensourcealgorithmsconsideredhereinbothbias and displacement error. These in-house algorithms use interpolationschemes that are utilised for subpixel interpolation within commercial DICalgorithmsandsowerelikelytobesuitablecandidatesforthisprocessingstepalso.Thebicubicmethodhasbeenshowntobesuperiorintermsofsinglepixelaliasingandsothismethodwouldberecommendedforthiscamerasystemandspecklepattern.Asthedemosaicingprocessdescribedhereisperformedoffline,increasing the order of the interpolation to biquintic splines may furtherimprovetheresults.ItcanbeseenfromthedecreasedintegerbiasseeninFigure6aandlowerLSEinFigure9 for themonochrome cameradatadue to thedemosaicing algorithms,that the nature of the speckle pattern can have a significant effect on the DICdata.Inthiscaseitisduetothesmoothingeffectofthedemosaicingalgorithmsontherawimageandsoproducingmoregradualchangesinintensity,whicharebettercharacterisedbytheinterpolatorusedintheDICalgorithm[5]Thedifferenceinpixelsize,andsoresolution,betweenthetwocamerasmakesdirect comparison more complicated than direct pixel-to-pixel mapping.However, this is a relevant question because these two cameras represent ahigh-resolution option from their respective genres. The two cameras have avery similar sized sensor and share the samemount, so these variables havebeenkeptconsistenteveniftheresolutionhasnot.Thiscombinedwiththefour-foldpricedifferencebetweenthetwocameras(attimeofwriting)inthecolourcamera’s favourshowthatthesetwocamerascouldbeconsideredcompetitorsfor the attention of a DIC practitioner. Consumer colour cameras such as theNikonD810asconsideredherearenotspecificallydesignedforDICandsotheyhavefeaturesthatwouldmakethemunsuitableformanyDICapplications.Largefilesizesandmovingpartsreducethemaximumusableframe-ratetotheregionof 1Hz, which lends them to quasi-static or long term testing. For long termtestingconsumercamerasbecomemoreattractiveduetothelowerinvestmentrequiredforasingletest.However,colourcamerascanonlybetakenseriouslyfor DIC if their performance and accuracy features are well understood. Thebicubicsplinedemosaicingalgorithmconsideredhereisasteptowardsthisgoal.

6 ConclusionColour cameras can be successfully used forDIC, but the demosaicingmethodusedtoproducethemonochromeinputimagecanhaveasignificanteffectontheDICresults.Bi-integerbias is the largestcauseoferrorforthesecolour imagesandthisisaprocessnotseenwithmonochromecameras.Thesingleintegerbiasseen in the colour camera is of the same order of magnitude as that seen inmonochromecameras,andinthiscasespecificallyitwasless.Thisisparticularlyrelevantasahistogramacrossasinglepixel isgenerallyusedtodeterminethepresence of integer bias for DIC. If thiswere performed for colour cameras it

wouldappearthattherewasverylittlebias,whenifthehistogramisextendedtotwopixelsthenaverystrongbiascanbeobserved.Toreducetheseeffectsasfaraspossible,abicubicsplinebasedinterpolationdemosaicingalgorithmhasbeenproposed. This algorithm has been shown to outperform all other consideredmethods in termsof aliasing and accuracy.More trials involving cameraswithvarying sensor sizes and resolutions are required before this could beconfidently recommended more widely, but the results shown here arepromisinginthatregard.

7 References[1] PetersWH,RansonWF.Digitalimagingtechniquesinexperimentalstress

analysis.OptEng1982;21:427–31.

[2] SuttonMA,SchreierH,Orteu JJ. ImageCorrelation forShape,MotionandDeformation Measurements: Basic Concepts,Theory and Applications.Springer;2009.

[3] HildF,RouxS.DigitalImageCorrelation:fromDisplacementMeasurementtoIdentificationofElasticProperties–aReview.Strain2006;42:69–80.

[4] Schreier H. W. SMAWYQBHA. Quantitative Error Assessment in PatternMatching: Effects of Intensity Pattern Noise, Interpolation, Strain andImageContrastonMotionMeasurements.Strain2009;45:160–78.

[5] Schreier HW, Braasch JR, Sutton MA. Systematic errors in digital imagecorrelationcausedbyintensityinterpolation.OptEng2000;39:2915–21.

[6] WienekeB,PrevostR.DICUncertaintyEstimationfromStatisticalAnalysisofCorrelationValues.Adv.Opt.MethodsExp.Mech.Conf.Proc.Soc.Exp.Mech.Ser.,vol.3,2014,p.125–36.doi:10.1007/978-3-319-00768-7.

[7] Yoneyama S, Morimoto Y. Accurate Displacement Measurement byCorrelation of Colored Random Patters. JSME Int J 2003;46:178–84.doi:10.1299/jsmea.46.178.

[8] White DJ, Take WA, Bolton MD. Soil deformation measurement usingparticle image velocimetry ( PIV ) and photogrammetry. Géotechnique2003;53:619–31.

[9] SakanashiY,GungorS,BouchardPJ.MeasurementofCreepDeformationinStainless Steel Welded Joints. Proc. SEM Annu. Conf. June 13-16, 2011Mohegan Sun, Uncasville, Connect. USA, vol. 5, 2011, p. 415–22.doi:10.1007/978-1-4614-0228-2.

[10] BouchardPJ,SakanashiY,GungorS.Spatiallyresolvedcreepdeformationofathicksectionstainlesssteelweldedjoint.3rdInt.ECCC-CreepFract.Conf.Rome,2014.

[11] BayerB.Colourimagingarray.3971065,1975.

[12] Gunturk BK, Glotzbach J, Altunbasak Y, Schafer RW, Mersereau RM.Demosaicking: Color filter array interpolation. IEEE Signal Process Mag2005;22:44–54.doi:10.1109/MSP.2005.1407714.

[13] DuboisE.Frequency-domainmethodsfordemosaickingofbayer-sampledcolor images. IEEE Signal Process Lett 2005;12:847–50.doi:10.1109/LSP.2005.859503.

[14] Kimmel R. Demosaicing: Image reconstruction from color CCD samples.IEEETransImageProcess1999;8:1221–8.doi:10.1109/83.784434.

[15] CorelCorporation.CorelPhotoPaintX62012.

[16] Hirakawa K, Parks T. Adaptive homogeneity directed demosaicingalgorithm.ImageProcessIEEETrans2003;3.

[17] MartinecE,LeeP.AMAZEdemosaicingalgorithm2010.

[18] GóźdźJ,RodriguezLS.Dcbdemosaicingalgorithm2010.

[19] HorváthG.eahddemosaicingalgorithm2010.

[20] Chang E, Cheung S, PanD. Color filter array recovery using a threshold-basedvariablenumberofgradients.Proc.SPIE,1999.

[21] LaVision.StrainMaster8.2.Goettingen:LaVisionGmbH;2014.

[22] FleetDJ,WeissY.OpticalFlowEstimation.Handb.Math.Model.Comput.Vision.,Springer;2005,p.239–58.

[23] Bouguet J-Y. Pyramidal Implementation of the Lucas Kanade FeatureTracker Description of the algorithm. 2000.doi:10.1016/j.tim.2005.08.009.

[24] MathworksInc.MATLAB2013a2013.

[25] Fincham AM, Spedding GR. Low cost, high resolution DPIV formeasurementofturbulentfluidflow.ExpFluids1997;23:449–62.

8 Figures

Figure1:SchematicofBayercolourfilterarray(withthe2x2pixelrepeatingunithighlightedwithdashed outline) showing how it is used in the in-house bilinear and bicubic spline demosaicingmethods.

Figure 2: Schematic diagram of the pixels knots the bicubic spline demosaicing method uses tointerpolatethetwounknowncolourchannels forthethreepossibleconfigurations: left forapixelwheretheredchannelvalueisknown,centreforapixelwherethegreenvalueisknownandrightforapixelwherethebluevalueisknown.Notethedifferenceinsampledareawheninterpolatingthegreenchannelincomparisontothegreenandredchannels.

Figure 3: Image of experimental setup, visible is the Nikon D810 colour camera, macro bellows,Nikon200mmf4macrolens,lightbox,fibreopticlightguideandrotationstagewithtarget.

a

bFigure4:a)Globalrotationmeasuredforeachcamerabyaveragingtheresultsofallanalysesfortherespective camera b) Absolute difference in global rotation measured using the differentdemosaicingalgorithmstothemeanforthecorrespondingcamera.

2 4 6 8 10 12 140

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

frame (#)

rota

tion

(rads

)

D810 averageDSQi2 average

2 4 6 8 10 12 140

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1x 10−4

frame (#)

abso

lute

rota

tion

diffe

renc

e fro

m m

ean

(rads

)

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear

Figure5:2DplotsofLSEintheydirectionof61x61subregionsizeanalysisusinga7pixelstepsizeforall theanalyses for the first4 rotation steps (entire images).Also included isa150x150pixelsectionofspecklepatternforeachdemosaicingalgorithmforcomparison.

a

bFigure 6: Histogram of the normalised per cent of vectors binned from -0.5 to +0.5 pixeldisplacementfora)11x11andb)61x61pixelsubregions,showingthepercentageofvectorsineachbinincomparisontotheidealfit.

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5−4

−3

−2

−1

0

1

2

3

4

displacement fraction (pixels)

Nor

mal

ised

per

cent

of v

ecto

rs

histogram for 11x11pixel subregion size

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5−4

−3

−2

−1

0

1

2

3

4

displacement fraction (pixels)

Nor

mal

ised

per

cent

of v

ecto

rs

histogram for 61x61pixel subregion size

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear

a

bFigure7:Histogramofthenormalisedpercentofvectorsbinnedfrom-1to+1pixeldisplacementfor a)11x11 and b)61x61 pixel subregions, showing the percentage of vectors in each bin incomparisontotheidealfit.

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−25

−20

−15

−10

−5

0

5

10

15

20

25

displacement fraction (pixels)

Nor

mal

ised

per

cent

of v

ecto

rs

histogram for 11x11pixel subregion size

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−25

−20

−15

−10

−5

0

5

10

15

20

25

displacement fraction (pixels)

Nor

mal

ised

per

cent

of v

ecto

rs

histogram for 61x61pixel subregion size

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear

a

bFigure8:Valuesfromthezerodisplacementfractionbinsofthehistogramsina)Figure7(binnedfrom-0.5to0.5pixeldisplacement)andb)Figure8(binnedfrom-1to+1pixeldisplacement).

0 10 20 30 40 50 60 70−0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

subregion size (pixels)

norm

alis

ed p

erce

nt o

f vec

tors

extent of 1x1 aliasing with subregion size

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear

0 10 20 30 40 50 60 70

0

5

10

15

20

25

subregion size (pixels)

norm

alis

ed p

erce

nt o

f vec

tors

extent of 2x2 aliasing with subregion size

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear

a

bFigure9:Averageacrossentireareaofall15imagesmeanleastsquareserrorforallcamerasanddemosaicingalgorithmsconsideredfordifferentsubsetsizesa)inpixelsandb)inmm.

0 10 20 30 40 50 60 700

0.01

0.02

0.03

0.04

0.05

0.06

0.07

subregion size (pixels)

mea

n LS

E (p

ixel

)

mean LSE variation with subregion size

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

1

2

3

4x 10−4

subregion size (mm)

mea

n LS

E (m

m)

mean LSE variation with subregion size

D810ahdD810vng4D810eahdD810dcbD810AMAZED810CorelD810BicubicD810LinearDSQi2DSQi2BicubicDSQi2Linear