SmartAlign for DigitalMicrograph - HREM Research · scan-distortion compensation plug-in (version...

April 2017

SmartAlign for DigitalMicrograph

Image Registration and Scan-distortion

Compensation Plug-in

SmartAlign Manual 1.2 (Provisional)

HREM Research Inc.

SmartAlignPlug-inUserManual

2

Preface

This usermanual describes how to install and operate the SmartAlign image registration and

scan-distortioncompensationplug-in(version1.x)forDigitalMicrograph.Thesoftwareisbased

onthefull-lengthmanuscriptbyDrLewysJonesandProfPeterD.Nellst:

Jones, L., et al.. “Smart Align—a new tool for robust non-rigid registration of scanningmicroscopedata”.AdvancedStructuralandChemicalImaging,18(2015).Link

As this tool applies post-processing steps to experimental data, users should state explicitly

whensuchstepshavebeenappliedbycitingtheabovepapersuchas:

“imagealignmentandscan-distortioncompensationwasperformedusingtheSmartAlignsoftwaredescribedin[1]”.

Mostofthismanualdescribestheprocessingofatomicresolutionimageseriesacquiredinthe

scanningtransmissionelectronmicroscope(STEM),howeverthesamestepsapplytodatafrom

the scanning tunnelingmicroscope (STM) and the atomic forcemicroscope (AFM).A separate

sectionattheendofthismanualdescribessettingsspecifictothesephysicalprobetechniques.

Contact

Enquiriesregardingsales,licensing,installationandoperationoftheplug-inshouldbedirectedto:

[email protected]

Forquestionsabouttheacademicmanuscriptreferencedabove,pleasecontactDrLewysJonesat:

[email protected]

CopyrightStatements

©Copyright2015-2016HREMResearchInc.

All rights reserved. This manual is protected by international copyright laws and treaties.Unauthorized reproduction and distribution of this manual, or any portion of it, will beprosecutedtothemaximumextentpossibleandmayresultinseverecivilandcriminalpenalties.

DigitalMicrographisatrademarkofGatanInc.


3

ContentsWhentousethisPlug-in.................................................................................................................5

Installation......................................................................................................................................6

SoftwareRequirements..............................................................................................................6

SoftwareInstallation...................................................................................................................6

TypicalWorkflow............................................................................................................................7

Plug-inMenu...................................................................................................................................8

SupportedFileTypes.......................................................................................................................8

DataAcquisition..............................................................................................................................9

‘Control’Tab................................................................................................................................9

AutoRigidAlignmentonFinish.................................................................................................11

‘Signal’Tab................................................................................................................................12

DataPreparation...........................................................................................................................13

DataShapingOptions................................................................................................................13

DataScalingOperations............................................................................................................14

ScanMetadataOptions.............................................................................................................14

RigidRegistration..........................................................................................................................16

SelectingtheAreaforRigidRegistration..................................................................................16

SelectingtheRigidRegistrationOptions...................................................................................18

Datapre-processingOptions.....................................................................................................18

RegistrationMethod.................................................................................................................19

OutputOptions.........................................................................................................................22

Non-rigidRegistration...................................................................................................................24

TheTypesofIterativeRegistrationCycles................................................................................24

TheNumberofIterativeCycles.................................................................................................25

TypesofNon-linearCycle..........................................................................................................25

SmoothingKernelSize...............................................................................................................26


4

RegistrationConvergence-theTarget,Acceleration&NumberofCycles...............................26

StoringtheTransformationData...............................................................................................27

ChoiceofReferenceImage........................................................................................................27

OutputOptions..........................................................................................................................28

TheProgressBar........................................................................................................................28

TheNROutputsandImageCropping........................................................................................29

AligningExtraImageSignals..........................................................................................................29

AligningSpectrumImageSeries....................................................................................................31

AdditionalMenuItems..................................................................................................................32

OutputDisplayOptions.............................................................................................................32

AdvancedOptions.....................................................................................................................32

SpecialCaseExamples...................................................................................................................34

DigitalSuper-ResolutionExample.............................................................................................34

ProcessingSTM&AFMData.....................................................................................................36

AdvancedFilteringforRigidRegistration..................................................................................38

Troubleshooting............................................................................................................................39

Auto-savefunctionnotworkingwithsomedetectorchannels.................................................39

References.....................................................................................................................................40


5

WhentousethisPlug-inTheSmartAlign plug-in forDigitalMicrograph is designed to correct themost common image

distortionsencounteredinatomic-resolutionimageseriesrecordedinthescanningtransmission

electronmicroscope (STEM), including image drift between frames and scan-distortionwithin

eachframe.Theseareshownschematicallybelow:

Imagedistortionfrequencydomains.Foradescriptionofthereferencesabove,pleaseseetheassociatedacademicmanuscript(Jonesetal.,2015).

SmartAlign is used to correct image-drift and scan-distortion. Scan-noise however, caused by

instabilitiesupinthekHzregime,canbecorrectedusingtheJitterbugplug-inalsoavailablefrom

HREMResearch(Jones&Nellist,2013).

Imageoffsetsbetweenframesinaseriesarereadilyidentifiable.Thesecanbeeasilycorrected

tosimplytheanalysisoftime-seriesorfocal-seriesdata.Scan-distortioncanappearasadditional

imagetransformations, includingaffinetransformations(stretch,skew/shear,smallrotations)

andnon-lineardistortions,causingotherwisestraightcrystallatticeplanestobecome‘wavy’.

Ifyourdataincludesthesetypesofshiftsand/ordistortions,thenitmaybepossibletorecover

some or all of the lost performance caused by scan-distortion. Of course if the image is very

heavilydegradeditwillnotbepossibletocompensatethisentirely.Equallythisplug-incannot

create resolution that did not exist prior to the incorporation of scan-noise. For further

informationandsomerelatedreadingpleaseseethereferencegivenonpage2.


6

Installation

SoftwareRequirements

ThefollowingisalistofthesoftwarerequirementsnecessarytoruntheSmartAlignplug-in:

• DigitalMicrograph(GATANTM)withDigiScanandEMControl• USBKeyDriver• HREMMouseToolPlug-in(Free-waredownloadablefromwww.hremresearch.com)• IPUPlug-in(Free-waredownloadablefromwww.hremresearch.com)• AcquireImageSeries(Free-waredownloadablefromwww.hremresearch.com)• MatlabMCR(MatlabCompilerRuntime)v8.2.

SoftwareInstallation

InstallingUSBKeyDriver

The user key driver should be installed by following the instructions given by the key driver

installer.Thekeydriverinstallercanbefoundonourwebsite.

InstallingSmartAlignPlug-in

Theplug-in canbe installedbydrag-and-drop copy to the folder “PlugIns” (ThePlugIns folder

shouldexistunderanormalinstallationoftheDigitalMicrograph.)

When the DigitalMicrograph is launched after placing the plug-ins into the PlugIns folder,

SmartAlignmenucommandswillappearunder“SmartAlign”menu.

InstallingMouseToolsPlug-in,IPUPlug-inandAcquireImageSeriesPlug-in

All the files relating these tool plug-ins, can be installed by drag-and-drop copy to the folder

“PlugIns.”(ThePlugInsfoldershouldexistunderanormalinstallationoftheDigitalMicrograph.)

When theDigitalMicrograph is launched after placing the plug-ins into the PlugIns folder, the

Mousetoolwillappearasanadditiontothestandardtools.

InstallingMatlabMCR

TheMatlabMCR should be installed by following the instructions given by the installer. The

MatlabMCRthatiscompatiblewiththeSmartAligncanbefoundonourwebsite.


7

TypicalWorkflowThe SmartAlign software is very flexible in how it is used, however, most tasks will follow a

similar workflow. These are summarized below and then discussed in more detail in the

followingsections.

l Thefirststageistherecording(orloading)ofanimageseries.

l Thisstagepreparestheimage-seriesforprocessing,isusedto trimtheseries,adjust thepixel-samplingandsetup the options corresponding to differentacquisitionconditions.

l Rigid-registration removes the translation offsetsbetween images in the series caused by stage /sampledrift.

l Non-rigidregistrationremovestheaffineand/ornon-lineardistortionswithineachframe.

l For drift measurement and compensation (optional)the lattice base vectors must be identified. Thesedirectionswillalsobeusedforandimageline-profilesused.

Theplug-in isdesignedsothateachstage intheprocessingcreatesanewdatawindowfor its

results. If for some reason the calculation fails, or the user wants to go back one stage and

choose a different setting, the new window can simply be closed and the result from the

previousstagewillstillbethere.

Thefollowingexampleswillusethesuppliedexampleimage-seriesofaSrTiO3nano-cubeedge

todemonstratethevarioussteps.

Dataacquisilon

Datapreparalon

Rigidregistralon

Non-rigidregistralon

SecondimageorSIcorreclon


8

Plug-inMenuThefigurebelowshowsthedefaultmenufortheSmartAlignplug-in:

SupportedFileTypesTobegin,theusershouldhavethefileopenwithintheDigitalMicrographworkspace.Toopena

fileeitheropenitfromyourcomputerorfromwithinDMbygoingtoFile>Open.DMsupports

images-seriesbeingsavedasasinglefile(asslicedata),howeveriftheimageswererecordedas

separate files, they can be imported using the ‘File’ > ‘Open Series…’ option to produce the

stackeddataonimport.

Theplug-insupportsallimageformatssupportedbyDM,however,itisstronglyadvisedthatthe

imagestobeprocessedhavereceivednoimagecompression(suchasJPEGs).


9

DataAcquisitionAtoolfortherecordingofseriesisincludedwithSmartAlign.

‘Control’Tab

DataName

Here you can enter a filename that will assigned to the recorded image series. If the user

chooses to create a stacked image (see next section), and use the auto-numbering and save

feature, each filename will be created using underscores to concatenate the fields with the

followingformat:

“DataName”+“_”+“SequentialNumber”+“_”+“DetectorName”

Forexample:

MySampleName_001_JEOLADF1


10

Ifyouchangetoanewsample/region/experimentyoucanchooseanew‘dataname’and

resetthenumberingbysimplytypingthenextdesirednumber.Numbersaregiveninathree

digitformat.

NumberofImages

This specifies the number of images in the series. After recording, the series can always be

trimmed. If the stability or damage tolerance of the sample is unknown, itmay be helpful to

overestimatethedesirednumberofframes. Ifdamageorcontamination isobserved,thenthe

usercanlaterdecidetorejectthelaterpartoftheseries.

In addition to specifying the number of images, the user should choosewhether towrite the

imagesindividuallytodiskortodisplaytheimagesasasinglestack(asslicedata).Ingeneral,a

single stacked image is more convenient. However, if recording very large images, or an

extremelylargenumberofframesitmaybepreferabletowritedirectlytodisk.

Inthecaseofcreatingastackedimage,theusercanfurtherselecttoauto-savethisfile.

Folder

Ifanauto-savingoptionwasselected,thenimageswillbesavedintothespecifiedfolder.Either

typetolocationdirectlyorselectthefolderusingthe‘Browse’button.

Magnification

Thiswillbeautomaticallyreadfromthemicroscopeandaddedtothe imageseriesmeta-data.

This doesnot need tobe editedunless theuser changed themagnification after opening the

acquisitionmenu.

ImageDimensions

This will be the height and width of the imagemeasured in pixels. The recorded images are

square.

PixelDwellTime

Thissetsthedwell-timeforeachpixel inunitsofmicroseconds.Thetimeforonescan-linewill

be the image widthmultiplied by this number. To adjust other time parameters such as the

flybackwaitingtime,usetheadvancedsettingsinthenormalSTEMDigiscandialoguewindow.

ImageRotation

Thissetstheoverallrotationofthescan,performingthesamefunctionas‘scan-rotation’inthe

Digiscansettings.Thescan-rotationofacquireddatahasaneffectontheeaseofprocessingthe

series.


11

For the rigid registration, the image alignment will make use of correlations calculated in

Fourier-space (more details later). To maximise the precision of this stage it is advisable to

ensurethatwhenrecordingimageseries,theFourierspots(includingtheircrossshapedstreaks

fromthefiniteimageborder)donotimpingeupononeanother.

For the non-rigid registration, if a low-order crystal lattice plane is aligned to the fast-scan

direction,thenoftensomescan-rowswillcontainonlyminimalcontrast.Asitisimagecontrast

whichisusedtoidentifyandcorrectforthescan-distortion,itispreferableforscan-rowstonot

bedevoidofcontrast.

Bothoftheseconditionscanbefulfilledbyaddingasmallscan-rotationoffofthecrystalplane

(say10°orso).

Afurtherrobustnesstosample/stagedriftcanbe incorporatedbyaddinganadditionalscan-

rotationsteppingof90°betweeneachimageframe.Inthiscasetheimageswillberotatedback

whenaddedtothestack.

AutoRigidAlignmentonFinish

Ifselected,anautomatedrigidregistrationcanbeperformedimmediatelyaftertheacquisition

hasfinished.Thiswillonlycorrectforimagetranslation(wholeframeoffsets),howeveritallows

theimagesintheseriestobesummedforinspection.Theusercanthencheck,forexample,the

sampletiltorimagedefocus.

Ifthisoptionisselected,thenonesignalcanbeselectedastheinspectionsignal.Othersignals

can be registered later. The ‘Setup’ button allows for the rigid-registration settings to be

adjusted.

To maximise workflow productivity, the outputs from this post-acquisition alignment can be

immediatelysavedtodisk,withtheoptiontoalsosavetheaveragethroughtheseriesasasingle

frame.


12

‘Signal’Tab

Onthistabtheusercanselectthesignalstoberecorded.Ifmultiplesignalsareselectedthese

willberecordedsimultaneously.Importantly,thismeansthatthesameprobe-offsetvectorswill

applytoeverysignal.

In this tab the number of signals available to the Digiscan, as well as their names, will be

automaticallydetected.


13

DataPreparationThedatapreparationstageshouldbeexecutedbeforeanyeitheroftheregistrationstages:

DataShapingOptions

Inthismenutheusercanchoosetoselectonlyasub-setoftheframesintheseries.Todothis

selectthecheckboxandthenchoosethefirstandlastframenumber(noteframecountinghere

isonebasedandnotzerobased).

The user can also select to reverse the frame order; this may be useful when processing or

comparingfocalseries.

Theusercanselecttotransposedata;thismaybeusefulifthedatahasbeenimportedfroman

FEI*.serfileorfromanotherpackagesuchasMatLab.


14

DataScalingOperations

NormalisingtheDataGrey-levelRange

FormostapplicationsthenumericalvaluesoftheSTEMimagepixelsarenotneeded.Inthiscase

the datamay be scaled (subtracting some constant and dividing by another) with no loss of

qualitativeinformationorrelativecontrast.

Therearetworeasonswhytheusermaywanttodothis;fordatawitha largeconstantoffset

(forexamplea largeamplifierblack-level) thedatawill havea largemeangrey-level. Thiswill

give a large DC offset in the Fourier based cross-correlations used for rigid registration.

Subtracting a constant reduces this offset term and may help with the rigid registration.

Additionally, the later non-rigid registration calculation is based on image intensity gradient

matchinganddividingtherangeofthedatabyaconstantmayacceleratethecalculationspeed.

Note,ifperformingaquantitativeanalysis,wherethepixelintensitiesdohaverealmeaning,this

optionshouldnotbeused.

DataResampling

This option allows the user to resample the data in x-y before any onward processing.

Resamplings smaller than one will reduce the number of pixels in the image (more coarse

sampling),andvalueslargerthanonewill increasethenumberofpixels inthedata(morefine

sampling).ReducingthesamplingmaybeusefulforcondensingdatafromlargeCCDcamerasor

to do initial quick calculations on less powerful computers. Increasing the sampling may be

usefultoachievedigitalsuper-resolution.Ingeneral,increasingthesamplingbymorethan,say,

three or four will not lead to any improvement but will slow the subsequent calculations

significantlyduetoincreasedmemoryusage.

Digitalsuper-resolutionisanadvancedtopicandaseparateexampleisgivenattheendofthis

manual.

ScanMetadataOptions

ScanRotationBetweenFrames

This option toggles different modes in SmartAlign depending on how the image series was

recorded. Specifically, it changes the ‘row-locking’ and ‘row-fitting’ behavior used in the non-


15

rigidregistrationoptions.Usersshouldseetheassociatedmanuscriptforafurtherdescriptionof

the‘row-locking’and‘row-fitting’behaviours.

Thedefault setting is ‘None’, thismeans thatno change in scan-rotationwasusedduring the

experimentalacquisition.Alternatively,SmartAlign canacceptdatawithmultiplesof90° scan-

rotation change between frames. For this data can be recorded as ‘orthogonal pairs’, that is

scanningat0°,90°,0°,90°…etc.,orasan‘orthogonalset,i.e.0°,90°,180°,270°,0°,90°…andso

on.

If either orthogonal type option is used, then the user should select whether the fast-scan

directionishorizontalorverticalinthefirstimageoftheseriesafteranytrimmingoftheseries

and/ortransposinghasbeenappliedfromtheearlieroptions.

Scan-lineTime

Heretheusercanspecifythetimeperfast-scanline.Thisisonlyusedtocalibratethefrequency

axisintheinstabilityfrequencyanalysis.

MedianEqualisation(TipHeightCorrection)

ThisisanadvancedoptionwhichattemptstocorrectforSTM/AFMtipheighterrors.

For STM/AFM image series exhibiting tip-height errors, the data can be adjusted in height to

bringeachscan-rowontoeitherarow-wisemedian,orglobalmedianheight.

The defaultmode for STEM or TEM imaging is ‘Off’. The only exception to this would be for

STEM data recorded with a cold-FEG gun where the emission fluctuation compensation has

failed(butthisisveryrare).


16

RigidRegistration

SelectingtheAreaforRigidRegistration

TheSmartAlign softwarecanperformthe rigid-registration inoneof twoways;eitheron the

entirefieldofview,oronaselected‘regionofinterest’(ROI).

Fullfield-of-view(withoutROI):

Tousethefullfieldofview,simplyselectthepre-processeddataasthefrontwindowandthen

select“Rigidregistration”fromtheSmartAlignmenu.

-à

Usingtheentireimagewillyieldanalignedimagewiththebiggestpossiblefieldofview,butit

maytakelongertocompute.Forlow-dosedata,wherecrystalfillsthefield-of-view,thenusing

thewholeareaislikelytoincreasealignmentprecision.


17

ROIAlignment

TousetheROIalignmentmode, firstdrawaROIonthepre-processeddatausingtheDMROI

tool:

YoushouldensurethattheROIislargeenoughtofullyencompassthefeatureofinterestinALL

slicesoftheseriesdata,notjustinthefirstslice.Youshouldinspecttheslicesbeforeproceeding

toensurethattheROIislargeenoughtoincludethefeatureaftersample/stagedrift.

à

Whereonlyoneregionofan image isof interest, thenusingaROIrigid-alignwillgenerallybe

faster.Thismayalsobeusefuliftryingtoalignverylargedatasetswherememoryusewouldbe

anissueandafiducialmarkispresent(seespecialFIB-SEMexampleatendofmanual).

AhybridoftheseapproachesistomeasuretheimageshiftsonthesmallerROIdatabutthento

correcttheimageshiftsonthefullfield-of-view;seethelatersectionon“AligningExtraImage

Signals”.


18

SelectingtheRigidRegistrationOptions

Eachexperimentaldatasetisunique;therigid-registrationmenucontainsoptionstoadjustthe

settingstogetthebestpossibleimagealignmentforanygivendataset.

Therigid-registrationmenuisdividedintothreesections:

Datapre-processingOptions

DataFiltering

If the image-series isnoisy then image filteringmay improve the reliabilityof the registration.

Two options are available; 3x3 median filtering and 5x5 Weiner filtering. Other advanced

filteringmaybeusedandthisspecialcaseisdiscussedwithaspecialexampleattheendofthis

manual.


19

DataWindowing

Forimageswhichhavecrystalfillingthewholefieldofview,crossshapesoftenappearcentered

ontheFourierspotsduetotheabruptedgesoftheimage.Thesecanaffectthereliabilityofthe

image correlations and windowing the images in real-space may improve this. A Hamming

windowcanbeselectedhere.

RegistrationMethod

RigidregistrationisachievedinSmartAlignusingcorrelationfunctions.Threespecificallynamed

correlationfunctions(betweentwoimagesgandh)areshownbelow:

Thecyclicalcross-correlation:

𝐶𝑜𝑟𝑟 𝑔, ℎ = 𝐹!! 𝐹 𝑔 ∘ 𝐹 ℎ ∗ (1)

Themutual-correlationfunction(MCF)(Heel,Schatz,&Orlova,1992):

𝑀𝐶𝐹 𝑔, ℎ = 𝐹!! ! ! ∘ ! ! ∗

! ! ∘ ! ! ∗ (2)

Thephase-correlationfunction(PCF)(Kuglin&Hines,1975):

𝑃𝐶𝐹 𝑔, ℎ = 𝐹!! ! ! ∘ ! ! ∗

! ! ∘ ! ! ∗ (3)

Moregenerallythesecanbeexpressedasacontinuumsetwheretheexponentonthe

normalising denominator, α, is a variable between zero and one(Kuglin & Hines, 1975).

Hereafter,forgeneralitythiswillbereferredtoasthenormalisedcorrelationfunction(NCF):

𝑁𝐶𝐹 𝑔, ℎ = 𝐹!! ! ! ∘ ! ! ∗

! ! ∘ ! ! ∗ ! (4)

Theneed toconsidermodifying thepoweron thenormalisingdenominator lies in the

rangeofspecimensthatmaybeneededtoanalysed.Forexample,specimenswithvery‘bland’

crystallography, suchasaconstant thicknessslabofSi[110],will require theregistration tobe

achieved entirely using the non-periodic information so a larger value of α would be

recommended (say 0.75). However, a free standing metallic nanoparticle free from

contamination can be registered very accurately considering the silhouette of this periodic

information,soalowervalueofαwouldbemostsuitable(say0.2).

Finally, if both the images being compared are noisy, the resultant correlation functions can

themselvesbenoisyandanadditionaloptiontofilterthesewithaButterworthfilterisprovided.


20

Oncetheuserhasselectedwhich filteringandcorrelation functionoptionswillbeused, there

are two further choices; how the offsets will be estimated and whether the offsets will be

refined:

EstimatingtheImageOffsets

Heretherearetwomodes,optimizedforbothhighsignal-to-noise(SNR)dataandlow-SNRdata.

Forthemajorityofdatasets,thehigh-SNRmodecanbeused,andonlyforthemostnoisydatais

itnecessarytousethelow-SNRmode.

Hign SNR Mode:

In the high-SNR mode it is assumed that the correlation functions setup with the previous

options will estimate the offset between images accurately but potentially imprecisely. To

mitigatethistherigid-registrationstageoftheSmartAlignplug-indoesnothaverelyonanyone

measurement to determine the offsets. Instead, when determining the image offsets, each

image can be analysed relative to its direct neighbours in the image series, then its second

nearestneighbours,third-nearestetc.

Heretheusercanselectthenumberofnearestneighbourstouse;theminimumisone,inwhich

caseeachimageisonlycomparedwiththepreviousone,upto(n-1),wherenisthenumberof

imagesintheseries.Tohelptheuserthemenuwillindicatethemaximumnumberthatcanbe

selectedalongsidetheoption.

Increasing this number excessively will increase calculation time with diminishing gains in

precision. Additionally, in the presence of gradual structural changes, such as beam-damage,

including ever further afar neighbours may reduce registration precision. A typical selection

wouldbethreeorfournearestneighbours.

Theexamplebelowshowstheresultofcomputingtheoffsetsforthefirst4nearestneighbours.


21

Thefigureshowsthreeplotsthehorizontaloffsetsbetweentheimages,theverticaloffsets,and

thesimilaritybetweenimagepairs.Eachvalueisplottedinthelowerhalf(triangle)ofthematrix

representingtheith-jthimages.Thesimilaritymatrixcanbeusefulindeterminingthenumberof

nearestneighbourstoinclude,ifthesimilaritybeginstodrop,thisindicatesthattheimagesare

nolongersimilarenoughandwouldnotimprove(orpossiblyworsen)theprecisionoftheoffset

estimation.

Oncethematricofneighbourdataiscompiledamutualsolutionisdetermined.

Low SNR Mode:

In“LowSNRMode”the imageseriesdatawill firstbecondensed intofoursubsetsto improve

the SNR of low-dose images. The sample drift between image 1 and image 5, should be

approximately the same as between image 2 and image 6, and so on. This gives redundant

observationsof the samedrift-rateandallows foramorepreciseestimate tobeachieved for

noisyimages.

After the four subsets have been analysed to determine the overall drift-rate, the offsets

betweenthesecorrectedfoursetsofdataareanalysedtocorrectforanyshiftsinducedbyany

90°imagerotationsused.

Together these two information describe all the shifts, the images are realigned and can be

refinedasdescribedinthefollowingsection.

Importantly,thismodeisonlyapplicabletoimage-seriesrecordedinasingleacquisition.Ifthe

seriesisgeneratedfromjoiningtogethertwoormoreshorterseries,thenthepriorknowledge

ofthemonotonicdriftbehaviourmaynotbevalidandthismethodwillnotbesuitable.


22

RefiningtheImageOffsets

Using the image-offsets estimated from theprevious stage the image series canbe realigned.

TheaverageofthisrealignedserieswillhaveahigherSNRthananyoneimagealone.Next,the

usercanreanalyseeachindividualimagerelativetothisaverageimage,andtheoffsetsnaybe

able tobe improved.Thisprocedure is somewhat iterativeand theuse candecidehowmany

attemptsshouldbemadetorefinetheoffsetestimates.

Atypicalvaluemightbetworefinements;selectingzerowillskipthisstage.

OutputOptions

Oncetheimageoffsetsarerefinedanewwindowwiththeresultswillbeoutput.

Whereshiftsexistbetweenimagestherearetwochoicestooutputthedata;firstly,thecanvas

ofthedatacanbeexpandedtoincorporatetheimagedrifts(resultsinabiggersizeimageseries),

or thedata canbe cropped toonly keep thephysical area common toall frames (results ina


23

smaller image series). The user can select which option they want with the “Crop output to

commonarea”checkbox.

If the user selects not to crop the data, theywill expand the canvas; in this case there is an

optionofwhattofilltheextraareawith.Thiscanbefilledwithblank(zeros)oritcanbefilled

withtheaverageofallnon-zeroobservationsfromthatposition.Theformeroptionisusefulfor

viewingtheimagedriftandobservingchangesintheilluminatedregionofthesample;thelatter

isusefulforamore‘panoramic’option,attemptingtomaximisethefieldofview.

In addition to the image series being output, the accumulated image is also shown after

processing. The final option in this section determines whether the average of the frames is

shown,orthesummation.

These rigid-registered results canbe savedat thispoint, it isnotnecessary tocontinue to the

non-rigidcalculation,asthiscanberesumedatalatertime.


24

Non-rigidRegistrationAfter finishing the rigid-registration,non-rigid registrationcanbeperformed.Beforebeginning

the non-rigid registration, if it is not already, the results from the rigid registration should be

selectedtobringitupasthefrontwindow.

Thismenuisdivideintothreesections,withoptionsrelatedtotheiterativeregistrationcycles,

thechoiceofreferenceimagetouse,andfortheoutputtingoftheresult.

TheTypesofIterativeRegistrationCycles

Since the SmartAlign algorithm does not enforce any prior knowledge on the expected

symmetryorappearanceofthesample,theregistrationrunsiteratively,convergingtowardsthe

ground truth. These iterations are broken into two types and can be selected separately or

together.


25

The first type is to correct the scan-distortion using only affine transformation. This type of

transformation is a ‘linear non-rigid transformation’. Linear, because any straight line feature

beforecorrectionwillbepreservedasastraightlineafterwards,andnon-rigidbecausedifferent

pixelscanmovebydifferentamountsinxandy.Usingthistypeofiterationtheusercancorrect

forimageshear,scale,and(small)rotations.

Aseverypixelinthefieldofviewisusedwhencalculatingtheaffinetransformforeveryframe

(6parameters),thistypeofcorrectionisveryrobusttoimagenoise.

The next type is the “Advanced non-rigid cycles”; these are the fully non-linear non-rigid

iterations. ‘Non-linear’, because now straight lines are no longer required to be preserved.

Unsurprisingly,non-lineartransformationisnecessarytocorrectnon-lineardistortions.

TheNumberofIterativeCycles

In the majority of cases few affine type cycles will be required. However, if scan-rotation

betweenframesisenabledorifthedriftrateofthestageisvaryingovertimethenmoremaybe

necessary.Fortunately, theseaffinecalculationsarecomputationallyveryefficientsoadefault

ofsay2-cyclesisagoodstartingvaluetotryinmostcases.

Thefullnon-linearcalculationismorecomputationallyintensive.Forquickcalculationstocheck

thequalityofdatatwocyclesmaybesufficient.Foramorefinalanalysisatypicalnumbermight

befourcycles.

TypesofNon-linearCycle

TheSmartAlignalgorithmoffersthreetypesofnon-lineariterativecycle.Thefirstoftheseisthe

so-called ‘scan-locked’ mode. This is described in more detail in the SmartAlign academic

manuscript. Inbrief, thismodemaximises theartefact robustnessof thealgorithmbyutilising


26

the prior knowledge that the STEM is a scanned instrument. Thismode is also applicable for

STM/ADF/SEMetc.Forserialmicroscopesthisisgenerallythemostappropriatemode.

In somecases,where the sample is rotating slightly in theplane (not tilting) then row-locking

willnotdescribethisadditionalmotion.Forthiscase‘row-fitting’shouldbeused.Thisallowsfor

the benefits of row-locking, but also allows an additional degree of freedom to describe the

rotation.Atypicalusecaseforthismodemightbenanoparticleslooselyattachedtoasubstrate.

The final mode available is the fully unconstrained mode. This is most suitable for parallel

acquireddatasuchasconventionalTEMandphotographicdata.

SmoothingKernelSize

Afterselectingthetypeofnon-linearcycle,theusermustspecifyasmoothingkernelsize.

There are two reasons why a smoothing kernel is used to constrain the diagnosed

transformationfields.Firstly;itisnotphysicallyrealisticforthediagnosedtransformationfields

tocontainabruptdiscontinuitiesandsoasmallsmoothingisjustifiedtoremovenoise.

Perhaps more importantly, and mostly when using the unconstrained (TEM) mode, it is

important that no crystallography is observed in the transformation fields. Remember, the

transformationfieldsareonlyadescriptionoftheenvironmentaldistortionsaffectingthedata.

As the distortions do not depend on the sample under observation, then these should not

exhibitsignsofcrystalperiodicity.

Rememberaddingasmallscan-rotation,andusingrow-lockingwillimproveSTEM(row-locked)

seriesprocessing.

RegistrationConvergence-theTarget,Acceleration&NumberofCycles

AstheSmartAlignsoftwaresolvesfortheimagedistortionsiteratively,successisdeterminedby

the measurement convergence. Three inter-related factors contribute to this convergence

behaviour.

TheConvergenceTarget

Theconvergencetargetspecifiestheexitcriterionfortheinnermostiterativeloop.Ithasunitsof

the average magnitude of pixel shift in the frame. The preset options range from weak-

convergence (averageshiftof10-4pixels) toa strongcriterionof10-6pixels.Theusercanalso

specifyacustomnumber.


27

TheConvergenceAcceleration

The convergence acceleration determines the aggressiveness of the iterative correction. A

highernumberwill increase calculation speedbut if thenumber is toohigh, and thedataare

noisy, then the convergence may become unstable. In this case a warning message will be

displayedandthesoftwarewillattemptto‘self-heal’buyreducingtheaccelerationby10%.

Theweaksettingyieldsanaccelerationof2throughtoavalueof20inthestrongsetting.The

usercanalsospecifyacustomvalue.

TheTotalNumberofIterativeCycles

Adding together the number of affine-only and non-linear registration cycles yields the total

numberofiterativecycles.Thefinalconvergencecriteriontargetspecifiedaboveisnotreached

immediately,butratherthestrictnessofthecriterion isapproachedgraduallyandreachesthe

ultimatetargetvalueatthefinalcycle.

StoringtheTransformationData

Ifmorethanoneimagesignalwasacquired,thentheprobe-offsetdatacanbetransferredfrom

onediagnosistocorrectothersignals.Ifthisisthecase,thentheusershouldcheckthe‘KeepNR

translationdata’box.Whenthistranslationdataisstoreditisstoredintheimagemetadatafor

lateruse,thedatacanbeusedtocompensateasecondsignal immediately,orthefilewithits

metadatacanbesavedandthetranslationsusedatalaterdate.Inthiscasethefilesizewillbe

largerbyapproximatelyafactorofthree(theoriginalimagedata,andtwonewmetadatawith

equaldimensionsdescribingtheshiftsinxandy).

ChoiceofReferenceImage

Theusercanchoosewhichreferencedataisusedasthecomparisonimage.

AverageofallFrames

Forthemajorityofcasesthebestreferencedatawillbetheaverageofallframesintheseries.In

thismodeeachimagewillbecomparedwiththeaveragewhendiagnosingtheshifts.Attheend

ofeachcycletheupdatedframeswillbeusedtomakeanewaveragereferenceimage.

FixedFrame

Analternativetousingtheaverageframeistospecifyaparticularframetouseasthereference

forallothers.


28

LocalAverageforInitialLoops

Athirdchoice,forspecialcases,istouseonlytheaverageoftheimagesimmediatelybeforeand

aftereachimageintheseriesasthereference–alocalaverage.Thislocalaveragecanbeuseful

if the image distortions are so large that averaging through thewhole serieswould cause an

excessivedegradationinreferenceimagequality.

Analternativeuseforthislocalaveragereferencemethodistheprocessingofdynamicmovies

suchasin-situdata,wheretheunderlyingsampleisgenuinelytransformingandanaverageofall

frameswouldnothaveaphysicalbasis. In this casesomedistortioncorrection is stillpossible

whilemaintainingtemporalresolution.

OutputOptions

Heretheusercanselecttoopentheresults inanewwindowandkeeptheoriginalopen(the

defaultbehaviour).Ingeneral,itisonlynecessarytodeselectthisincomputerswithverylimited

memoryorwithverylargedatasets.

Theusercanalsochooseherehowtooutputthecompoundimageafterprocessing.Ingeneral,

the‘average’imageisthemostsuitable(includingforfractionalADForSTM/AFMheightdata),

buta‘summation’imagemaybeusefulifthedatarepresentscountsorsomeotherquantityyou

wishtoaccumulate.

TheProgressBar

After choosing all the appropriateoptions, pressing “OK”will start the calculation.During the

calculationaprogressbarwillbeshown,aswellassomemoredetailedtextbasedreadoutsin

theOutputwindow.

The calculation can be aborted if necessary by pressing the “Cancel” button. In this case a

confirmationdialogwillbedisplayed:


29

In this caseonly thecurrently runningnon-rigid calculationwillbeaborted.Theexisting rigid-

registeredresultwill remain in theworkspace.TheNRcalculationcanbestartedagain,or the

rigidresultssavedatthispointandthecalculationcontinuedlater.

TheNROutputsandImageCropping

AftertheNRregistrationiscomplete,anddependingontheseverityofthedriftanddistortions,

pixelsshiftedfromoutsidethescannedareawillappeararoundtheedgeoftheframeasgrey

pixels. Itmaybenecessary to croponeor twopixels fromaround theperimeter. This canbe

donebyselectintheROItoolfromthetoolbarandthenselectinthe‘Crop’commandfromthe

SmartAlignmenu.

AligningExtraImageSignalsAfter an alignment has been performed (either rigid or non-rigid), then this can be further

applied toanother image signal so longas thedatawereacquired simultaneously. If thedata

werenotacquiredsimultaneouslythenthedriftsandprobedistortionswillnotbeequalandthis

isnotapplicable.

Todothisselect‘AlignSecondSignal…’fromthemainmenu.


30

HereforexamplethediagnosisfromtheregistrationofanADFseriesiscopiedtoanABFseries.

AfterpressingOK,boththenewalignedsignalwillbedisplayedaswelltheaverageofthenew

series.

RigidRegisteringAdditionalSignals

Inthecaseofrigidalignment,ifthealignmentwasperformedonareducedROI,thentheuser

willbeaskedwhethertousethese imageshiftsforthewhole image,ortoagainreturnonlya

reducedROI:

If the user selects “the same area” then the same area as the ROI used on the first signal is

returned,iftheuserselects“thewholeimage”thenthewholeimageisalignedandcroppedto

thelargestpossiblecommonarea.

Non-rigidRegisteringAdditionalSignals

InthecaseofNRregistration,itisonlypossibletoreturnthesamefield-ofview,andtheabove

choicewillnotbepresented.


31

AligningSpectrumImageSeriesIf the image series which has been aligned represent the simultaneous images frames

accompanyingspectrumimageacquisitionthentheprobeoffsetscanbecopiedfromtheimage

seriestotheseriesofspectralvolumes.

First all the steps above should be completed; during rigid alignment cropping should be

selected, and during non-rigid registration the option to keep NR translation data should be

selected.

With the aligned image series selected as the frontwindow, select ‘Align SI Data…’ from the

SmartAlignmenu.

Afileexplorerwindowwillopenandyoushouldselectallthespectralvolumesthatmatchthe

image series. The spectral volume series will be read into DM in the order based on their

filenames. For example adding numbers or letters to the SI filenameswill help thembe read

correctly.Theexamplehereshowsan8-frameimageseriesand8spectrumimagesselected.

Thesizeofthespectrumimagesinthethirddimension(numberofchannels)willalsobe

displayed,andifnecessaryareducedrangecanbereconstructedtominimiseRAMusage.

Youshouldverifythatthecorrectreferencedataisselectedandpress‘OK’.

While the SI realignment and NR registration is running, an integrated spectral view will be

shownaswellasanintegratedspatialview.Thesearenotinteractiveandarejustvisualisations

tomonitortheprogress.Oncecompletethecorrectedspectralvolumewillbedisplayed.


32

AdditionalMenuItemsTherearetwofurthertabscontainingoutputdisplayoptionsandtheadvancedsettings.

OutputDisplayOptions

Thistaballowstheusertodisplaymoreorfeweradditionaldiagnosticfiguresduringprocessing.

Theseoptionsincludeachoiceforthegranularityoftextmessageoutputs(displayedintheDM

resultswindow)duringprocessing,aswellastheadditionalgraphicaldisplayofvariables.

ThesesettingsaresavedwhentheregistrationisbegunandwillbepreservedafterDMisclosed

forthenexttime.

AdvancedOptions

Thismenucontainssettingsthatrarelyneedtobechanged.


33

Theseoptionscanbeusedtosetanupperlimitforthemaximumnumberofiterationperimage

in the innermost correction loop. Reducing this numbermay increase calculation speedbut if

reducedtoofarmaystopthecalculationfromconvergingfully.

Inthecaseofanimageserieswithafeaturelessbackground,thenconstrainingtheedgesofthe

transformationfieldmayimprovealgorithmconvergence.

Inthecaseofverynoisydata,asmallsmoothingcanbeappliedtotheindividual imagesorto

theaveragereferenceimage.

Thenoiseconstantcontrolstherobustnesstoextremeoutlierimagepixels.Alargernumberwill

increaserobustnessbutattheexpenseofreducedsensitivity.Thisnumbershouldingeneralnot

bechanged.

Thedebugmodesettingisonlyusedforremotesupportuse.


34

SpecialCaseExamples

DigitalSuper-ResolutionExample

Consider the exampleADF image-series belowof a perovskite grain boundarywith 16 frames

with512x512imagepixelseach.Thishasbeenrecordedathigh-resolutionbutwithinsufficient

pixel-sampling.

TheFFTshowsspotsarevisiblerightouttotheNyquistlimit,andthehighestobservablespotis

the(220)spacing(approx..1.38Å.A64x64pixelROIisenlargedonthefarright–again,nothow

thesamplingistoolow.Ordinarilythiswouldbeapoorchoiceofsettingtorecordtheimage;we

wouldusuallyincreasethenumberofpixelsorreducethefieldofview(increasemagnification).

However,boththeseoptionswouldincreasethetotalelectron-doseoverthesample.

With thisdata,angle-measurement, structurematching,peak-findingor strain-mappingwould

beverychallenging.Toovercomethis,wewillincreasethesamplingofthisframe-seriesinpost-

processingusingdigitalsuper-resolution.

Note: this is notoptical super-resolution, itwill not extend the resolution beyond that of the

originalSTEMprobe,butratherthis isdigitalsuper-resolutionwheretherestoredimage-series

averagewillcontainmorediscernibleinformationthatanyoneframeatthelowsampling.

Todothisselect“Resampledata”inthe“Preparedata”menu.Forthisexample,wewilluse3x.

Nowthedatashouldberigid,andnon-rigidregisteredasnormal.

The resultsbelowshowthe lower-rightpartof the imageafter resamplingandalignment, the

reducedFourier transform(central50%),andacroppedenlargementof thesameregionas in

theearlierfigureabove.


35

Afterprocessingthe(331)spotisvisible(approx.0.9Åresolution),andthedetailintheinterface

become clear. The recovered resolution is likely slightly poorer than had the image been

recordedunderoptimalsamplingconditions,butitisanimprovementontherawdata.

Note: thedata-resamplingmustbedonebefore the rigidandnon-rigid registration toachieve

thedigitalsuper-resolution.Doingitafterwardwouldjustbeasimpleimagescaling.

Belowshowsfurtherenlargementsofthebefore(single-frame)andafter(average-frame)data.

Goingbeyond3x data-resamplingwill likely offer little benefit andwillmostly just add to the

calculationtime.


36

ProcessingSTM&AFMData

For many scanned probe microscopy (SPM) datasets, it is often necessary to first remove a

planarrampbackground.ThistoolisprovidedasautilityinSmartAlignandcanbedoneatany

point.Howeverforbestresults,itisrecommendedtoperformthisstepatthestartofthedata

processing.

With the data loaded and stacked to create a series, select “Remove Image Ramp” from the

mainSmartAlignmenu:

Ifasingleimageisselectedthiswilloperateonce,ifaseriesisselected,everyimageintheseries

willhavearampcalculatedandremoved.

Theresultwillbesimilartotheexamplebelow:


37

Note:therange(heightdata)intheimageispreserved,andtherampissubtractedsuchthatthe

newimageminimumiszero.

Aftertherampremovaliscomplete,youcancontinuetoprocessthedataasusual.


38

AdvancedFilteringforRigidRegistration


39

Troubleshooting

Auto-savefunctionnotworkingwithsomedetectorchannels.

Onsomesystemsyoumayseethefollowingerrorwhentryingtousetheauto-savingfeatureforsomedetectorchoicesettings:

Thisiscausedwhenthedetectornameitselfcontainsaslash(“/”)charactercausingthefilenametotrytoaccessanon-existentdirectory.Tocorrectthis,gototheDigiScanSetupmenu,select“Advanced”>“AnalogueInput”andeditthenameofthedetectorcausingtheerror;inthiscase“GatanBF/DF”canbechanged,forexample,to“GatanBF-DF”.


40

ReferencesHeel,M.van,Schatz,M.,&Orlova,E.(1992).Correlationfunctionsrevisited.Ultramicroscopy,46,

307–316.

Jones,L.,&Nellist,P.D.(2013).IdentifyingandCorrectingScanNoiseandDriftintheScanning

Transmission Electron Microscope. Microscopy and Microanalysis, 19(04), 1050–1060.

http://doi.org/10.1017/S1431927613001402

Jones,L.,Yang,H.,Pennycook,T.J.,Marshall,M.S.J.,VanAert,S.,Browning,N.D.,…Nellist,P.

D.(2015).SmartAlign—anewtoolforrobustnon-rigidregistrationofscanningmicroscopedata.

AdvancedStructuralandChemicalImaging,1(1),8.http://doi.org/10.1186/s40679-015-0008-4

Kuglin, C.D.,&Hines,D. C. (1975). Thephase correlation image alignmentmethod.Proc. Int.

Conf.onCyberneticsandSociety,1975,163–165.

Date post:	12-Jul-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

SmartAlign for DigitalMicrograph - HREM Research · scan-distortion compensation plug-in (version...

Documents