J Anim Ecol. 2018;87:533–545. wileyonlinelibrary.com/journal/jane | 533© 2017 The Author. Journal of Animal Ecology © 2017 British Ecological Society
Received:11April2017 | Accepted:17October2017DOI:10.1111/1365-2656.12780
R E V I E W
A computer vision for animal ecology
Ben G. Weinstein
DepartmentofFisheriesandWildlife,MarineMammalInstitute,OregonStateUniversity,Newport,OR,USA
CorrespondenceBenG.WeinsteinEmail:[email protected]
HandlingEditor:LauraPrugh
Abstract1. Acentralgoalofanimalecologyistoobservespeciesinthenaturalworld.Thecostandchallengeofdatacollectionoftenlimitthebreadthandscopeofeco-logical study.Ecologistsoftenuse imagecapture tobolsterdata collection intime and space. However, the ability to process these images remains abottleneck.
2. Computervisioncangreatlyincreasetheefficiency,repeatabilityandaccuracyofimage review. Computer vision uses image features, such as colour, shape andtexturetoinferimagecontent.
3. Iprovideabriefprimeronecologicalcomputervisiontooutlineitsgoals,toolsandapplicationstoanimalecology.
4. I reviewed187existingapplicationsofcomputervisionanddividedarticles intoecologicaldescription,countingandidentitytasks.
5. Idiscussrecommendationsforenhancingthecollaborationbetweenecologistsandcomputer scientists and highlight areas for future growth of automated imageanalysis.
K E Y W O R D S
automation,cameratraps,ecologicalmonitoring,images,unmannedaerialvehicles
1 | INTRODUCTION
Observing biodiversity can be expensive, logistically difficult andtime-consuming.Manyanimalsarerare,secretiveandinhabitremoteareas. Animal presence and behaviour may vary over broad spatialand temporal scales, anddependon importantbut infrequentlyob-servedevents,suchasbreeding,predationormortality.Directobser-vationof these events can be disruptive towildlife, andpotentiallydangerous to observers.To reduce cost, labour and logistics of ob-servation,ecologistsareincreasinglyturningtogreaterautomationtolocate, countand identifyorganisms innaturalenvironments (Pimmetal., 2015). While image capture has greatly increased sampling,our ability to analyse images remains a bottleneck in turning thesedataintoinformationonanimalpresence,abundanceandbehaviour.Computervisioncanincreasethebreadth,durationandrepeatabilityofimage-basedecologicalstudiesthroughautomatedimageanalysis(Delletal.,2014;Kühl&Burghardt,2013;Pennekamp&Schtickzelle,
2013).Computervision is a formof image-based computer sciencethatusespixelvaluestoinferimagecontent(LeCun,Bengio,&Hinton,2015).Theatomicunitofdata in computervision is an imagepixelthat represents colour in the visible spectrum. Pixels are arrangedintogroupssuchthatpixelproximity,orientationandsimilaritycreatea group identity. Pixelvalues, and the resulting group identity,maychange among images to create a sequence of objects. By creatingrulesforthepixelcharacteristics,relationshipsandchangesthroughtime, computervisionalgorithmscan replace laborioushand-reviewofecologicalimages.
Thegrowthinecologicalimagedataisfuelledbyitseconomy,ef-ficiencyand scalability (Bowley,Andes,Ellis-Felege,&Desell, 2017;Dell etal., 2014). Massive repositories of image data are availablefor ecological analysis, uploaded from field-based cameras (Giraldo-Zuluaga,Gomez,Salazar,&Diaz-Pulido,2017;Swansonetal.,2015;Zhang,He,Cao,&Cao,2016)orcapturedbycitizenscientists(Deselletal., 2013;Joly etal., 2014). For example, researchgradedatasets
534 | Journal of Animal Ecology WEINSTEIN
from iNaturalist (675,000 images of 5,000 species,VanHorn etal.,2017) and Zooniverse (1.2million images of 40 species; Swansonetal.,2015),highlightthegrowthinhigh-qualityimagescapturedbyresearchersandthepublic.However,imagedatacollectionhasgreatlyoutpacedimageanalysistools.Whileahumanmaybebetteratfind-inganimals in time-lapsevideo (Weinstein,2015),orhaveagreaterknowledgeofbirdidentification(Bergetal.,2014),whenconfrontedwith100,000images,itisdifficulttofindthetime,organizationandconcentrationtovalidateeachimagemanually.Myaimistodescribetheongoingworkinutilizingcomputervisionforanimalecology,pro-videabriefdescriptionoftheconceptsthatunitecomputervisional-gorithms, and describe areas for collaboration and growthwith thecomputervisioncommunity.
2 | APPLICATIONS OF COMPUTER VISION TO ANIMAL ECOLOGY
Ecological computer vision has grown out ofmultiple disciplines,with contributions from computer science (Branson, Van Horn,Belongie,&Perona,2014),astronomy(Arzoumanian,Holmberg,&Norman,2005)andremotesensing(LaRue,Stapleton,&Anderson,2016).This article covers applicationsof computer vision to find,count and study animals in natural landscapes using images col-lected in the human visual spectrum.Applications from specimenmorphometrics, microscopy (Pennekamp & Schtickzelle, 2013)andanimaltracking in laboratorysettingsarereviewedelsewhere(Dell etal., 2014; Robie, Seagraves, Egnor, & Branson, 2017). Tofindarticles,IusedWebofSciencetosearchfor“ComputerVisionAND(EcologyORAnimals),”yielding284articles.Ithenperformedthreeadditionalsearchesforarticlesusingimageanalysistools,butlacking the computer vision label: “Automated species measure-mentAND images” (n=103), “Automated speciesdetectionANDimages” (n=126) and “Automated species identificationAND im-ages”(n=196).Finally,Ireviewedthefirst200resultsfromGoogleScholarfor“ComputerVisionANDecology”publishedsince2000.For all searches, articles were included based on the followingcriteria.
1. The article described a peer-reviewed application of computervision. Articles introducing hardware for image capture, or re-viewing existing applications, were excluded.
2. Thearticlewasaimedatansweringanecologicalquestion,broadlydefinedas the identity,demographyandbehaviourofanimals innatural environments using images collected in human visualspectrum.
3. Theapplicationusedanautomatedorsemi-automatedimageanal-ysis algorithm. Articles using manual review of images wereexcluded.
Thissearchandfilteringcriteriaresultedin187articles,withconsis-tentgrowthincomputervisionapplicationsovertime(Figure1).Thesearticlesusedavarietyofopensourcetoolstoaidimageanalysis(Table1).
Iorganizedarticlesaroundthreecommontasksforecologicalcomputervision:description,countingandidentification(Figure2).Fromtheper-spectiveofimage-basedcomputervision,descriptionisthequantifica-tionofthecoloration,patterningandrelativesizeofanimalsandtheirimmediatesurroundingenvironment.Countingisthedetectionandenu-merationofanimalswithinanimage.Identityistheclassificationofanindividualorspeciesbasedonitsappearance.Foreachofthesetasks,mygoalistohelpecologistsgraspthecurrentpossibilityforimageau-tomationbyintroducingbasicterminology,applicationsandhighlightingacasestudy.
3 | DESCRIPTION
Ecologistoftenseektounderstandanimalappearanceandtheir re-lationshiptothesurroundingenvironmentusingdigitalobservations.Thesecretivenatureofmanyanimalsmakesdirectdescriptiondisrup-tiveandpotentiallydangeroustoboththeorganismandresearcher.Computer vision algorithms have greatly increased the ability tonon-invasively measure organisms through image analysis (n=56).Toascertainthesize,positionandspectralcharacteristicsofecologi-calobjects in images,computervisiontoolsuse imagefeatures (seeBox1)tofindimportantpixelswithinandamongimages.Imagefea-turesareoftenareasofhighturnoverinpixelvalues,causedbyedgesofobjectsofinterest.Forexample,tocorrectlyoutlineaflyingbird,algorithmsmight look for theareaswhere thewings intersectwiththesky(Atanbori,Duan,Murray,Appiah,&Dickinson,2016).Imagefeatureshavebeenprimarilyusedtostudytheevolutionaryecologyof animal coloration (Stoddard, Kilner, & Town, 2014), shape (Lavyetal.,2015)andpatterning(Levy,Lerner,&Shashar,2014).Comparedto human review, computer vision provides amore consistentwaytoscoreanimalappearanceacross imagesbyusingnon-RBGcolourspaces, such asHSVorYChCr,which are less sensitive to changesin illumination and other image artefacts (Kühl & Burghardt, 2013;Troscianko,Skelhorn,&Stevens,2017).Bycomparingimagefeatures,computer vision can be used to study animal camouflage (Tankus& Yeshurun, 2009) and biomimicry (Yang, Wang, Liang, & Møller,2016).Forexample,Stoddardetal.(2016)developededgedetectionalgorithms toevaluate the relative camouflageofnesting shorebirdspeciesascomparedtotheirnestingsubstrate(Figure3b).
Imagefeaturescanalsobeusedtomeasuresizeinbothspecimensand free-living animals (Olsen &Westneat, 2015). Based onmulti-ple images frompairs of cameras, computervision tools have beenusedtodescribeanimalsizeandshape,suchasinwhales(Howland,Macfarlane, & Tyack, 2012), and coral (Jones, Cantin, Berkelmans,Sinclair, & Negri, 2008; Naumann, Niggl, Laforsch, Glaser, &Wild,2009).Thenextfrontierfor image-basedecologicaldescription is in3Dreconstructionofmorphologyandmovement (Haggag,Abobakr,Hossny,&Nahavandi,2016;Lavyetal.,2015).Three-dimensionalim-aginghasrecentlybeenusedtotrackanimalbehaviourwithinlargein-doorenclosures(e.g.Barnardetal.,2016),andapplyingthesetoolstoanimalsinnaturallandscapesisandevelopingareaofresearch(Robieetal.,2017).
| 535Journal of Animal EcologyWEINSTEIN
F IGURE 1 Thegrowthincomputervisionapplicationsovertime(n=187).Fromtheperspectiveofimage-basedcomputervision,descriptionisthequantificationofimagefeaturestodescribecoloration,patterningandrelativesizeofanimalsandtheirsurroundinghabitat.Countingisthedetectionandenumerationofanimalswithinanimage.Identityistheclassificationofanindividualorspeciesbasedonitsappearance
TABLE 1 Commonlyusedtoolsforcomputervisionapplicationtoecology
Name Reference Task Comments
OpenCV Bradski(2000) Description,Counting,Identity
Sourcelibraryforcomputervisionalgorithmsinpython/java/C++
ImageJ Abràmoffetal.(2004) Description,Counting Segmentationandthresholding
BISQUE Kvilekvaletal.(2009) Description,Counting Alsoservesasahostingplatformforimageanalysistools
AgisoftPhotoscan – Description Commercialsoftwarefor3Dmodelreconstructionfromimages
StereoMorph OlsenandWestneat(2015) Description Rpackagefor3dreconstructionandimagecalibration
NaturePatternMatch Stoddardetal.(2014) Description Comparingfeaturesamongecologicalimages
MotionMeerkat Weinstein(2015) Counting Backgroundsubtractionforanimaldetectioninvideosandimages.
GoogleCloudAPI – Identity ClassificationofimagecontentusingCloudVisionAPI,deeplearningsourcelibraryusingTensorFlow
Merlin VanHornetal.(2015) Identity BirdidentificationappforiPhoneandAndroid
Wildbook Cralletal.(2013) Identity Individualidentificationanddatamanagementtools
536 | Journal of Animal Ecology WEINSTEIN
3.1 | Case study: High- resolution mapping of penguin colonies using structure- through- motion
To map habitat suitability, ecologists often use remotely sensed en-vironmentalvariablesasaproxy for theenvironmentalconditionsen-counteredbyanimals.Whiletraditionalremotesensingcapturescoarse
changesinhabitatquality,animalsexperiencetheenvironmentatfine-scales,inthreedimensions,andfromalandscapeperspective.McDowalland Lynch (2017) generated ultra-fine scale (<1cm)maps of penguincoloniesbystitchingtogetherthousandsofoverlappingimagesusingatechniquecalledstructure-from-motion.Theresultingthree-dimensionalsurfaceallowedfine-scalemappingofGentoopenguin(Pygoscelis papua)
F IGURE 2 Thenumberofecologicalcomputervisionarticles(n=187)foreachfocaltaxaandcomputervisiontask.Fromtheperspectiveofimage-basedcomputervision,descriptionisthequantificationofimagefeaturestodescribecoloration,patterningandrelativesizeofanimalsandtheirsurroundinghabitat.Countingisthedetectionandenumerationofanimalswithinanimage.Identityistheclassificationofanindividualorspeciesbasedonitsappearance
Box 1 Glossary of key computer vision terms for ecological image analysis
Description
Features:Pixelpropertiesbasedonthecolour,texture,orrelationshiptosurroundingpixels.
Colour space:Numericsystemusedtodescribethespectralinformationcontainedinpixelvalues.
Edges:Imagelocationswithabruptchangesinpixelvalues,alsoknownas‘corners’.Oftenusedtofindcorrespondingpointsbetweenimages.
Structure-from-motion:Approachforreconstructingthe3Dstructureofastationaryobjectbasedonstitchingtogetherimagestakenfrommultipleangles.
Optical flow:Theidentificationofanalogouspixelsamongimages,usedtotrackobjectorcameramovement.
Counting
Segmentation:Theprocessofpartitioningimagesintolabelledregions.
Contours:Curvedlineswhichencompassconnectedpixelswithsimilarcolour,intensityortexture.
Blobs:Groupsofconnectedpixelswithafixedidentityorlabel.
Image morphology:Imageprocessingtoolsformanipulatingpixelsbasedonthevaluesofthesurroundingpixels.Forexample,‘opening’reducesnoiseintheforegroundbyremovingweaklyconnectedpixels.
Background subtraction:Theremovalofirrelevantcontentestimatedfrommultipleframesofvideo.Subtractingthestaticportionsoftheframefromthecurrentimageyieldstheestimatedforegroundobjects.
Identity
Labelled training data:Imageswithknownobjectsofintereststhatcanbeusedtotrainmachinelearningclassifiers.
Unsupervised classification:Multidimensionalclusteringalgorithmstodividepixelsintoanapriorinumberofgroupsbasedonimagefeatures.
Neural-network or ‘deep learning’:Ahierarchicalmachinelearningclassifierthatusestrainingdatatocategorizeimagecontentwithoutapriorispecificationofimagefeatures.
| 537Journal of Animal EcologyWEINSTEIN
nests andcapturedvariation in slopeandaspect thatmayhavebeenmissedbycoarsersatellite-basedremotesensing(Figure3a).
4 | COUNTING
While remotely placed cameras provide a low-cost alternative tohumanobservers,theamountofdatageneratedbyfieldstudiescanbeoverwhelming.Thepotentiallyhighcostofimagereviewandstor-agemeansthatfindingtheanimalsofinterestwithinlargebatchesofimages can improve the speed and efficiency of biodiversitymoni-toring.Evenmotion triggeredcamera trapssuffer frommany false-positive images due to wind and moving vegetation. In computervision,findingnovelobjectswithinseriesofimagescanbeachievedusingbackgroundsubtraction,whichdistinguishessedentaryobjects,suchastreesandclouds,frommovingobjects,suchasanimals,withinvideosorgroupsofimages(PriceTacketal.,2016;Ren,Han,&He,2013;Weinstein,2015) (Figure4a).Abackgroundmodel is created
bycomputinganexpectedimagebasedonthepreviouspixelvalues(Stauffer&Grimson,1999).Theforegroundmodeldescribesthenon-backgroundpixelsasafunctionofthedifferencebetweentheprevi-ousbackgroundmodelandthecurrentframe(Figure4a;Christiansen,Nielsen, Steen, Jørgensen, & Karstoft, 2016; Sobral & Vacavant,2014).Thebackgroundmodelchangesovertimebasedonnewpixelvalues, thereby reducing false positives from shifts in illuminationandexternalmovement,suchaswind,wavesorcamerashake.Onceimages have been divided into foreground and background pixels(knownassegmentation),objectsarepartitionedintodiscretegroups,withconnectedsetsofpixelscorrespondingtoindividualorganisms.
Ifound55articlesthatusedaformofbackgroundsubtractiontode-tectandcountanimals,primarilyformammals(n=24)andbirds(n=22).Thesestudiesreporthighaccuracyinremovingemptyframes,buttherewerepersistentchallengesinreducingfalsepositivesfromstrongwindandotherextraneousmovementinheterogeneousenvironments(PriceTacketal.2016).Tailoringdetectionalgorithmsto individual taxacangreatly improve accuracy, for example, Zeppelzauer (2013) reported
F IGURE 3 Applicationsofcomputervisiontodescribingecologicalobjects.(1)FromMcDowallandLynch(2017),athree-dimensionalmapofthePortLockroypenguincolonywascreatedbyoverlayinghundredsofindividualphotographs(1a)todescribethelocationofGentoopenguin(Pygoscelis papua)nests(1b).Flagsdenoteoccupiedpenguinnestsidentifiedintheimages.Thesurfacewasturnedintodigitalelevationmap(1c)tomeasuretherelativepositiveandhabitatchoicebyindividualpenguinsfornestsiteselection.(2)FromStoddardetal.(2016),snowyplover(Charadrius nivosus)nestclutch(2a)segmentedintoeggandbackgroundregions(2b),edgedetectionwasusedtoquantifyedges(2c),inordertocalculatethedegreeofeggcamouflagecomparedtothebackgroundsubstrate(2d).SeeAcknowledgementsforcreditsandpermissions [Colourfigurecanbeviewedatwileyonlinelibrary.com]
(a) (b) (c)
(a) (b) (c) (d)
538 | Journal of Animal Ecology WEINSTEIN
>95% accuracy in detectingAfrican elephants (Loxodonta cyclotis) bybuildingacolourmodelfromtrainingdata.State-of-the-artapproaches(Renetal.,2013;Zhangetal.,2016)canbothidentifyimagesofinterest,aswellasdefinewherewithinanimageananimaloccurs.Thisisacrucialfirststepincroppingphotostoanalysespeciesidentity(seebelow).
New computer vision tools have opened new avenues for imagedatacollection.Automatedcountdatahavebeentakenfromtime-lapsevideo (Steen&Ski, 2014), camera traps (Matuska,Hudec,Kamencay,Benco,&Zachariasova,2014),uploadedbycitizenscientists (Kosmalaetal.,2016)andcapturedfromairbornesensors(vanAndeletal.,2015).In particular, automateddetection algorithms are increasingly used tofind large animalswithin remotely sensed imagery captured by high-resolutioncommercial satellites (Barber-Meyer,Kooyman,&Ponganis,2007)andunmannedaerialvehicles(Hodgson,Kelly,&Peel,2013;Liu,Chen,&Wen,2015;vanAndel etal., 2015).Commercial satellite im-ageryofferswidespatialcoverageatsub-metre resolution,but is lim-ited by atmospheric conditions, temporal coverage and high cost. Tofindanimalswithinthisimagery,studieshaveusedpixel-basedanalysis(Fretwell,Staniland,&Forcada,2014), imagedifferencing(LaRueetal.,2015)andsupervisedclassificationusingmachinelearning(Yangetal.,2014).Severalapplicationsfocusonaggregationsofindividualsincolo-nialbreedingsitesduetotheirlargespatialsizeanddistinctvisualsigna-tureonthesurroundingenvironment(Barber-Meyeretal.,2007;Lynch,White,Black,&Naveen,2012).WhileresultsfromSouthernrightwhales(Eubalaena australis)(Fretwelletal.,2014),polarbears(Ursus maritimus)(LaRueetal.,2015),andsavannaungulates(Yangetal.,2014)highlightthepromiseof this technology,considerableautomation isneeded to
reduce the laborious handvalidation of images at scale (LaRue etal.,2016).
Incomparison to satellite-based imagery,unmannedaerialvehicleshavetheadvantagesofgreatertemporalflexibilityandlowcost(Seymour,Dale,Hammill,Halpin,&Johnston,2017).Thetrade-offisthedecreasedspatial extent limited by flight time and legal restrictions (Crutsinger,Short,&Sollenberger,2016).UAVshavebeensuccessfullyusedtocountwaterbirdpopulations,duetothebirds’openhabitatandcolonialbreed-ingstrategy (Descamps,Béchet,Descombes,Arnaud,&Zerubia,2011;Groom, Krag Petersen, Anderson, & Fox, 2011). Chabot and Francis(2016)reportedthatautomatedcountsofwaterbirdswerewithin3%–5%ofhumancountsacross16applications.RecentimprovementsofUAV-basedcountingincludeutilizinghyperspectraldata(Beijboometal.,2016;Witharana&Lynch,2016),pixel-shapemodelling (Liuetal.,2015)andcombiningbackgroundsubtractionwithmachinelearning(Torneyetal.,2016)(Figure4b).Recenteffortstocountanimalsusedeeplearningneu-ralnetworksarepromising,butrequiretensofthousandsoftrainingim-agesgatheredbyhumanannotation(Bowleyetal.,2017).
4.1 | Case study: Counting hummingbird–plant interactions using background subtraction
Topredict therules thatdeterminethe interactionsamongspecies,ecologistsoftenusethefrequencyofinteractionsasaproxyforfit-ness effects (Bartomeus etal., 2016). To determine the number ofvisitsbetweenbirdsandflowers,WeinsteinandGraham(2017)usedtime-lapse cameras to filmmultiple days of flower visitation.Using
F IGURE 4 Applicationsofcomputervisiontodetectingandcountingecologicalobjects.(a)Backgroundsubtractionofvideoframesyieldsthedesiredmotionobject(Weinstein,2015)basedonchangesinpastpixelvalues.(b)CountingwildebeestfromimagerycapturedbyunmannedaerialvehicleinTanzania(Torneyetal.,2016).Theleftpanelarecorrectidentificationsofwildebeest,therightpanelarefalsepositivescausedbyaflockofjuvenileostrich.SeeAcknowledgementsforcreditsandpermission[Colourfigurecanbeviewedatwileyonlinelibrary.com]
| 539Journal of Animal EcologyWEINSTEIN
background subtraction algorithms, theywere able to process over8,000hr of hummingbird visitation videos (Weinstein, 2015). Thisvideo-based strategy allowed sampling at much greater temporalextents, and thereforeminimized thepotential foroverlooking rareinteractions.Fromthesedata,theauthorsmodelledspeciesinterac-tionsbasedonmorphologicalsimilarityandflowerabundancetotestpredictionsofoptimalforagingtheory(Weinstein&Graham,2017).
5 | IDENTITY
Ecologistsoftenneedtoinventorythediversityoftaxaorthenumberofindividualsofagivenspeciesinageographicarea.Thestrongre-lationshipbetweensamplingdurationandobservedspeciesrichnessmeans that data collection can often be expensive and logistically
challenging.Image-basedanimalclassificationhasthepotentialtore-ducecosts,allowgreatergeographiccoverageandcauselessdistur-bancetopotentiallysensitiveecosystems.
For individual-level identification, computervision algorithms useimagesofknownindividualstomatchnewimagesbasedonthesimilar-ityofphenotypicpatterns(Figure5a).Bymatchingtheimagefeaturesamongimages,matchingalgorithmsscorethe likelihoodthattwoim-agesareofthesameindividual.Foranimalswithuniquemarkings,thiscanbea low-costalternative toexpensive trappingand taggingpro-grams.Thisapproachwaspioneeredforfluke identification inmarinemammals (Adams,Speakman,Zolman,&Schwacke,2006;Beekmans,Whitehead, Huele, Steiner, & Steenbeek, 2005; Gilman, Hupman,Stockin,&Pawley,2016)andhassincebeenappliedonawiderangeoftaxa,fromzebras(Equus grevyi)(Crall,Stewart,Berger-Wolf,Rubenstein,& Sundaresan, 2013), to elephants (L. cyclotis) (Ardovini, Cinque, &
F IGURE 5 Applicationofcomputervisiontopredictingindividualandspeciesidentity:(a)Matchingalgorithmsscorethesimilarityofphotographedzebrastoalibraryofknownimagestotrackindividualsovertime(Cralletal.2013).(b)FromMarburgandBigham(2016),adeeplearningclassifieristrainedonastarfishspeciesclassbasedontrainingdataoflabelledimages.Theclassifieristhenusedtopredicttestingdatatoevaluatetheaccuracyoftheapproach.Inthisexample,thetrainingandtestingdataareseparateobjectswithinthesameimageframebasedonboundingboxesthatdistinguishanimalsfromtheimagebackground.SeeAcknowledgementsforimagecreditsandpermissions[Colourfigurecanbeviewedatwileyonlinelibrary.com]
540 | Journal of Animal Ecology WEINSTEIN
Sangineto, 2008), and box turtles (Terrapene carolina) (Cross, Lipps,Sapak,Tobin,&Root,2014).Thesemethodsareeffectiveinidentifyinganimalswithcomplexmarkings,suchasgiraffes(Giraffa camelopardalis)(Bolger,Morrison,Vance,Lee,&Farid,2012),whalesharks(Rhincodon typus)(Arzoumanianetal.,2005)andcatfish(Rineloricaria aequalicuspis)(Dala-Corte,Moschetta,&Becker,2016),and range fromcompletelyautomated(Town,Marshall,&Sethasathien,2013),toinvolvinghumanfeedback duringmatching (Duyck etal., 2015). Crall etal. (2013) re-portedaccuracyratesrangingfrom95%forGrey’szebras(E. grevyi)to100%forjaguars(Panthera onca)usingtheHotSpotteralgorithm,whichcanbeaccessedthroughtheWildbookwebplatform.
Automatedspeciesidentificationisrapidlydevelopingfieldwithanex-plosionofnewapproachesandpromisingresults(Figure5b).Whileinitialattemptsfocusedontraditionalmachinelearningwithanaprioridivisionofimagefeatures(e.g.Blanc,Lingrand,&Precioso,2014;Lytleetal.,2010),the accuracy of these approacheswas generally low (>70%).However,recentadvancesusingnewdeeplearningmodelshavegreatlyimprovedmodel performance across a wide variety of animal taxa, from coral(Beijboometal.,2016)to largemammals (Gomez,Diez,Salazar,&Diaz,2016) (Table2).Themajorityofapplications I reviewedhadaparticulargeographicfocus,forexampletherodentcommunityoftheMojavedes-ert(Wilberetal.,2013).Thenextstageisageneraltestofmachinelearn-ingmodelsacrosssystemstofindtheoptimalnumberoftrainingimages,modelparametersandtherequiredspectraldiversityofpotentialclassesthatleadstoincreasedpredictiveperformance(VanHornetal.,2015).
5.1 | Case study: Merlin, a bird identification app powered by deep learning neural networks
The Merlin project demonstrates the potential for revolutionarychangeinecologicalidentification(Farnsworthetal.,2013).Merlin
is the joint project from Visipedia and the Cornell Laboratory ofOrnithology to identify600commonNorthAmericanbirdspecies(VanHornetal.,2015).The identificationalgorithmusesGoogle’sTensorFlowdeep learningplatform,aswellascitizensciencedatafromeBirdtogeneratepotentialspecieslistsgivenauser’slocation(Bransonetal.,2010).WhileMerlinisprimarilygearedtowardsciti-zenscientists,pairingthistechnologywiththegrowingnumberofpublicallyaccessiblephotos(e.g.iNaturalist.org)promisestobolsterobservationsofrareandcrypticspeciesforbiodiversitymonitoring.
6 | COLLABORATION WITH COMPUTER VISION RESEARCHERS
Thecombinationofhigh-qualitydata,appliedusecasesandinterestingproblemswillleadtoproductivecollaborationsamongecologistsandcomputervisionresearchers.Whilecomputervisiontoolsarebecom-ingmoreaccessibletoecologists,state-of-the-artsolutionswillbenefitfromcollaborationwiththecomputervisioncommunity.Findingandmaintainingthesecollaborationscanbedifficultgiventhedifferenceinterminologyandaimsofecologistsvs.computerscienceresearchers.Isuggesthighlightingthreeareasofpotentialmutualinterest:
1. Ecology has intriguing and challenging technical problems. Thenatural world is complex and heterogeneous. Changes in illu-mination and backgrounds make animal detection difficult.Changes in organism appearance and shape are challenging forclassification algorithms. Ecologists should emphasize the gen-erality of their proposed problem, and frame collaborations asa potential area for development of new algorithms, ratherthan as an applied example.
ReferenceTraining images Taxa
Species or classes
Average accuracy (%)
Wilberetal.(2013) 5,362 Mammals,Reptiles 7 76.4
Yuetal.(2013) 22,533 Mammals 18 83.8
Chen,Han,He,Kays,&Forrester(2014)
9,530 RainforestMammals
19 38.3
Hernández-Sernaetal.(2014)
1,800 Fish,Butterflies 32 92.87
92 11 93.25
Atanborietal.(2016) – Birds 7 89.0
Bergetal.(2014) Avg.of200perspecies
Birds 500 66.6
Beijboometal.(2016) 28,400 Coral 10 88.9
Marburg&Bigham(2016) 8,586 Benthicinverte-bratesandfish
10 89.0
Gomezetal.(2016) 14,346 Savannaanimals 26 88.9
Qin,Li,Liang,Peng,&Zhang(2016)
22,370 Fish 23 98.6
Villonetal.(2016) 1,400 Fish 8 65.8
Sunetal.(2017) 9,160 Fish 15 77.27
Fengetal.(2016) 4,530 Moths 50 53.12
TABLE 2 Evaluationstatisticsforrecentcomputervisionapplicationstopredictingspecies-levelidentity.Articlesshownonlyincludeapplicationstomorethanfivespeciesandquantifiedtheclassificationaccuracyusingatestingdataset.Accuracyisonlyreportedforthebestperformingmodelineachpaper.Articlesareorderedbypublicationdate
| 541Journal of Animal EcologyWEINSTEIN
2. Ecologists can improvecomputervisionalgorithmsbyprovidingbiological context (Berg etal., 2014). Ecological rules that mayseemtobecommonsense,suchas“thereshouldnotbefishonland,”requireneuralnetworkstoidentifybothobjectsinanimage(“fish”),scenecontext(“land”)andtherelationshipbetweenimagefeatures.Onewaytoovercomethiswouldbetocumbersomelytrainanalgorithmwiththousandsofimagesoflandwithoutfish.Ecologyprovidesamorestraightforwardandeffectivemethodbyusingimagemetadata,suchastimeorlocation,toassistinimageclassification.Forexample,combiningimagelocationwithexpertvettedspeciesregionalchecklistsmightshowthatonlyafewspe-cies with a given coloration occur in given location. Similarly,imagecontextcanassistfuturepredictions.Forexample,ifanal-gorithmidentifiesawildebeestinanaerialimage,itmaybemorelikelytoalsofindzebras(Swansonetal.,2015).Finally,ecologicalcontextcanreducetheburdenofgatheringtrainingdatabyex-ploitingtheinherentconservationofbodyplansamonganimalstocreatehierarchicallabels.Ratherthanthinkingofallpotentialin-dividualcategories(e.g.blackbear,grizzlybear,polarbear,etc.),hierarchical labellingexploits theconnectionsamonganimals tocreatenestedcategories(e.g.Ursus).Tree-basedclassificationap-proacheshavebeeneffective inotherareasofcomputervision,
andfitsnaturallywiththestudyofevolutionarytaxonomy(Favret&Sieracki,2016).
3. Ecologistsarecollectingvastamountsoflabelleddata.Computervision applications, and especially deep learning approaches, re-quire significant training and testing data. High-quality datasetsaredifficulttofind,andalackoflabelleddataisamajorobstacleincomputer vision research (Belongie&Perona, 2016;Berg etal.,2010;Gomezetal.,2016).Packagingimagedatasetsandmakingthempubliclyavailablewill raiseawarenessof theopportunitiesforecologicalcollaboration.
7 | FUTURE GROWTH
Thefutureofecologicalcomputervisionwillcombinenewalgorithms,data and collaborations to study animals in natural environments.The rise of neural networks as the central tool in image classifica-tion(Gomezetal.,2016;LeCunetal.,2015),backgroundsubtraction(Christiansenetal.,2016)andimagedescription(Mohanty,Hughes,&Salathé,2016) is akeydevelopment thatwithbringnewoppor-tunitiesforecologicalcomputervision(Figure6).Untilrecently,thegrowthofthesetoolshasbeenslowedbyalackofaccesstocutting
F IGURE 6 Overviewofaneuralnetworkformachinelearningprediction:(a)Pixelconvolutionscreatecombinationsofinputpredictorsbydownsamplingandpoolingimagefeatures,(b)agenericdeeplearningstructure,inputdatapassesthroughhiddenlayers,callednodes,tocreatepathwaysfrompredictorstoprediction.Theactivationscoreateachofthesenodesisusedtoestimatemodelweights.Incurrentdeeplearningapplications,therewillbemanyhiddenlayersofnodestocreatecombinationsofinputpredictors[Colourfigurecanbeviewedatwileyonlinelibrary.com]
542 | Journal of Animal Ecology WEINSTEIN
edgealgorithms.TherecentunveilingoftheGoogleCloudMachineLearningplatformcouldbeaquantumleapinaccessforecologists.Releasedin2016,GooglegivesusersaccesstoawebservicetoretrainmodelsusingGoogle’spopularTensorFlowsoftware.TensorFlowisacomputationalgraphalgorithmthatrepresentsmathematicalopera-tionsasnodesandstoresdatainmultidimensionalarrays.Ratherthanbuildingamodelfromscratchforeachapplication,userscanretrainpre-builtmodelstoaddnewimageclasses.Knownastransferlearn-ing,thisapproachusesthestrengthsoftheunderlyingarchitecture,butaddsflexibilityforspecializedproblems.Thisgreatlyreducesthetimeandexpertiseneededtoimplementimageanalysissolutions.
Apersistentchallengeincomputervisionapplicationsiscollectingsufficientlabelleddata(Bergetal.,2010).Newdatacollectionoppor-tunities through datamining (Zhang, Korayem, Crandall, & Lebuhn,2012)andcitizenscientistswillbroadenthepotentialsourcesof la-belled ecological data (Swanson, Kosmala, Lintott, & Packer, 2016).Thenaturalexcitement forplantsandanimalsmeansthatgatheringfurtherlabelleddataispossiblethroughonlinecitizenscientistefforts(VanHornetal.,2015).Inparticular,projectsontheZooniverse,iNat-uralistandWildlife@homewebplatformsprovideawayofengagingimportantusercommunities(Deselletal.,2013;Kosmalaetal.,2016).Thenextstepisintegratingcitizenscientistsasapartofgreaterau-tomation, ratherasanalternative toautomation.Knownas “humanin loop”approaches,thisstrategycan learndirectlyfromhumanan-notations to provide feedback and recommendation for future datalabelling(Bransonetal.,2010;Reda,Mateevitsi,&Offord,2013).Thiswillcombinetheexpertiseandexcitementfromcitizenscientists,withthegreaterstandardizationofautomation.
ACKNOWLEDGEMENTS
Iwouldliketothankmanyresearchersfortheircontributionstothisreview: JasonParham (Figure5a), andAaronMarburg (Figure5b)contributedunpublishedfigures.FiguresfromTorneyetal.(2016)andStoddardetal. (2016)wereusedunderthePlosOneCreativeCommons License. Thanks to Shanis Barnard, DrewWeber,WillPearse, Margaret Kosmala, Tanya Berger-Wolf, Chuck Stewart,J. J. Valletta and Oscar Beijboom for their helpful input. Themanuscript was improved by the assistance of two anonymousreviewers.Themetadata for the reviewed filesaswell as the re-sults of each of the fourWeb of Science searches are providedin theSupporting Information.Theauthordeclaresnoconflictof interest.
DATA ACCESSIBILITY
Data available from the Dryad Digital Repository: https://doi.org/10.5061/dryad.b700h(Weinstein,2017).
ORCID
Ben G. Weinstein http://orcid.org/0000-0002-2176-7935
REFERENCES
Abràmoff, M. D., Magalhães, P. J., & Ram, S. J. (2004). Image process-ing with imageJ. Biophotonics International, 11, 36–41. https://doi.org/10.1117/1.3589100
Adams, J. D., Speakman, T., Zolman, E., & Schwacke, L. H. (2006).Automating image matching, cataloging, and analysis for photo-identification research. Aquatic Mammals, 32, 374–384. https://doi.org/10.1578/AM.32.3.2006.374
Ardovini,A.,Cinque,L.,&Sangineto,E.(2008).Identifyingelephantphotosbymulti-curvematching.Pattern Recognition,41,1867–1877.https://doi.org/10.1016/j.patcog.2007.11.010
Arzoumanian, Z., Holmberg, J., & Norman, B. (2005). An astronomicalpattern-matchingalgorithmforcomputer-aidedidentificationofwhalesharks Rhincodon typus. Journal of Applied Ecology, 42, 999–1011.https://doi.org/10.1111/j.1365-2664.2005.01117.x
Atanbori, J., Duan, W., Murray, J., Appiah, K., & Dickinson, P. (2016).Automatic classification of flying bird species using computer vi-sion techniques. Pattern Recognition Letters, 81, 53–62. https://doi.org/10.1016/j.patrec.2015.08.015
Barber-Meyer,S.M.,Kooyman,G.L.,&Ponganis,P.J.(2007).Estimatingthe relativeabundanceofemperorpenguinsat inaccessiblecoloniesusing satellite imagery. Polar Biology, 30, 1565–1570. https://doi.org/10.1007/s00300-007-0317-8
Barnard,S.,Calderara,S.,Pistocchi,S.,Cucchiara,R.,Podaliri-Vulpiani,M.,Messori, S.,&Ferri,N. (2016).Quick, accurate, smart: 3Dcomputervision technology helps assessing confined animals’ behaviour.PLoS ONE,11,1–20.https://doi.org/10.1371/journal.pone.0158748
Bartomeus,I.,Gravel,D.,Tylianakis,J.M.,Aizen,M.,Dickie,I.,&Bernard-Verdier,M.(2016).Acommonframeworkfor identifyingrulesacrossdifferenttypesofinteractions.Functional Ecology,10,1–40.https://doi.org/10.1111/1365-2435.12666
Beekmans,B.W.P.M.,Whitehead,H.,Huele,R.,Steiner,L.,&Steenbeek,A.G.(2005).Comparisonoftwocomputer-assistedphoto-identificationmethods applied to spermwhales (Physeter macrocephalus).Aquatic Mammals,31,243–247.https://doi.org/10.1578/AM.31.2.2005.243
Beijboom,O.,Treibitz,T.,Kline,D.I.,Eyal,G.,Khen,A.,Neal,B.,…Kriegman,D.(2016).Improvingautomatedannotationofbenthicsurveyimagesusingwide-bandfluorescence.Scientific Reports,6,23166.https://doi.org/10.1038/srep23166
Belongie,S.,&Perona,P.(2016).Visipediacirca2015.Pattern Recognition Letters,72,15–24.https://doi.org/10.1016/j.patrec.2015.11.023
Berg,T.,Liu,J.,Lee,S.W.,Alexander,M.L.,Jacobs,D.W.,&Belhumeur,P.N. (2014).Birdsnap: Large-scale fine-grainedvisual categorizationofbirds.Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,pp.2019–2026.
Berg,T.L.,Sorokin,A.,Forsyth,D.A.,Hoiem,D.,Endres, I.,&Farhadi,A.(2010).It’sallaboutthedata.Proceedings of the IEEE,98,1434–1452.https://doi.org/10.1109/JPROC.2009.2032355
Blanc,K.,Lingrand,D.,&Precioso,F.(2014).Fishspeciesrecognitionfromvideo using SVM classifier. Proceedings of the 3rd ACM International Workshop on Multimedia Analysis for Ecological Data – MAED ‘14, pp.1–6.NewYork,NY:ACMPress.https://doi.org/10.1145/2661821
Bolger, D. T., Morrison, T. A., Vance, B., Lee, D., & Farid, H. (2012). Acomputer-assisted system for photographic mark-recapture anal-ysis. Methods in Ecology and Evolution, 3, 813–822. https://doi.org/10.1111/j.2041-210X.2012.00212.x
Bowley,C.,Andes,A.,Ellis-Felege,S.,&Desell,T.(2017).Detectingwild-life in uncontrolled outdoor video using convolutional neural net-works. Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016,pp.251–259.
Bradski,G.(2000).TheOpenCVLibrary.Dr Dobbs Journal,25,120–126.Branson, S., Van Horn, G., Belongie, S., & Perona, P. (2014). Bird spe-
cies categorization using pose normalized deep convolutional nets.
| 543Journal of Animal EcologyWEINSTEIN
British Machine Vision Conference (BMVC). http://www.bmva.org/bmvc/2014/files/paper071.pdf
Branson,S.,Wah,C., Schroff, F.,Babenko,B.,Welinder,P.,Perona,P.,&Belongie, S. (2010). Visual recognition with humans in the loop. InDaniilidis K., Maragos P., Paragios N. (eds) Computer Vision - ECCV 2010. ECCV 2010. Lecture Notes in Computer Science, vol 6314.Berlin,Germany:Springer.
Chabot,D.,&Francis,C.M. (2016).Computer-automatedbirddetectionandcounts inhigh-resolutionaerial images:Areview.Journal of Field Ornithology,87,343–359.https://doi.org/10.1111/jofo.12171
Chen,G.,Han,T.X.,He,Z.,Kays,R.,&Forrester,T.(2014).Deepconvolu-tionalneuralnetworkbasedspeciesrecognitionforwildanimalmon-itoring. IEEE International Conference on Image Processing (ICIP),2014,858–862.
Christiansen,P.,Nielsen,L.,Steen,K.,Jørgensen,R.,&Karstoft,H.(2016).DeepAnomaly:Combiningbackgroundsubtractionanddeeplearningfordetectingobstaclesandanomaliesinanagriculturalfield.Sensors,16,1904.https://doi.org/10.3390/s16111904
Crall,J.,Stewart,C.,Berger-Wolf,T.Y.,Rubenstein,D.L.,&Sundaresan,S.R.(2013).Hotspotter–Patternedspeciesinstancerecognition.Applications of Computer Vision (WACV), 2013 IEEE Workshop,pp.230–237.
Cross,M.D., Lipps,G.J., Sapak,J.M.,Tobin,E.J.,&Root,K.V. (2014).Pattern-recognition software as a supplemental method of identify-ingindividualeasternboxturtles(Terrapene c. carolina).Herpetological Review,45,584–586.
Crutsinger,G.M.,Short,J.,&Sollenberger,R.(2016).ThefutureofUAVsin ecology: An insider perspective from the SiliconValley drone in-dustry. Journal of Unmanned Vehicle Systems,4,161–168.https://doi.org/10.1139/juvs-2016-0008
Dala-Corte,R.B.,Moschetta,J.B.,&Becker,F.G.(2016).Photo-identificationasatechniqueforrecognitionofindividualfish:Atestwiththefresh-waterarmoredcatfishRineloricaria aequalicuspisReis&Cardoso,2001(Siluriformes:Loricariidae).Neotropical Ichthyology,14,e150074.
Dell,A.I.,Bender,J.A.,Branson,K.,Couzin,I.D.,dePolavieja,G.G.,Noldus,L.P.J.J.,…Brose,U.(2014).Automatedimage-basedtrackinganditsapplication in ecology.Trends in Ecology and Evolution,29, 417–428.https://doi.org/10.1016/j.tree.2014.05.004
Descamps,S.,Béchet,A.,Descombes,X.,Arnaud,A.,&Zerubia,J.(2011).Anautomaticcounterforaerialimagesofaggregationsoflargebirds.Bird Study,58,302–308.https://doi.org/10.1080/00063657.2011.588195
Desell,T.,Bergman,R.,Goehner,K.,Marsh,R.,Vanderclute,R.,&Ellis-Felege,S. (2013).Wildlife@Home: Combining crowd sourcing and volunteercomputing to analyze avian nesting video. Proceedings - IEEE 9th International Conference on e-Science, e-Science 2013,pp.107–115.
Duyck, J., Finn, C., Hutcheon,A.,Vera, P., Salas, J., & Ravela, S. (2015).Sloop:A pattern retrieval engine for individual animal identification.Pattern Recognition,48,1055–1069.
Farnsworth,E.J.,Chu,M.,Kress,W.J.,Neill,A.K.,Best,J.H.,Stevenson,R.D.,…Pickering,J.(2013).Next-generationfieldguides.BioScience,63,891–899.https://doi.org/10.1525/bio.2013.63.11.8
Favret,C.,&Sieracki,J.M.(2016).Machinevisionautomatedspeciesiden-tificationscaledtowardsproductionlevels.Systematic Entomology,41,133–143.https://doi.org/10.1111/syen.12146
Feng, L., Bhanu, B., & Heraty, J. (2016). A software system for auto-mated identificationand retrievalofmoth imagesbasedonwingat-tributes.Pattern Recognition,51, 225–241.https://doi.org/10.1016/j.patcog.2015.09.012
Fretwell, P. T., Staniland, I. J., & Forcada, J. (2014).Whales from space:Countingsouthernrightwhalesbysatellite.PLoS ONE,9,1–9.https://doi.org/10.1371/journal.pone.0088655
Gilman, A., Hupman, K., Stockin, K. A., & Pawley, M. D. M. (2016).Computer-assisted recognition of dolphin individuals using dorsalfin pigmentations.2016 International Conference on Image and Vision Computing New Zealand (IVCNZ),1-6.
Giraldo-Zuluaga, J.-H., Gomez, A., Salazar, A., & Diaz-Pulido, A. (2017).Camera-trap images segmentation using multi-layer robust principalcomponentanalysis.arXiv:1701.08180
Gomez,A., Diez, G., Salazar,A., &Diaz,A. (2016).Animal Identificationin LowQuality Camera-Trap Images UsingVeryDeep ConvolutionalNeural Networks and Confidence Thresholds. Advances in Visual Computing: 12th International Symposium, ISVC 2016, Las Vegas, NV, USA, December 12–14, 2016, Proceedings, Part I,pp.747–756.SpringerInternationalPublishing.https://doi.org/10.1007/978-3-319-50835-1
Groom,G.,KragPetersen,I.,Anderson,M.D.,&Fox,A.D.(2011).Usingobject-basedanalysisofimagedatatocountbirds:MappingofLesserFlamingosatKamfersDam,NorthernCape,SouthAfrica.International Journal of Remote Sensing,32,4611–4639.https://doi.org/10.1080/01431161.2010.489068
Haggag,H.,Abobakr,A.,Hossny,M.,&Nahavandi,S.(2016).Semanticbodyparts segmentation for quadrupedal animals.2016 IEEE International Conference on Systems, Man, and Cybernetics,pp.855–860.
Hernández-Serna, A., & Jiménez-Segura, L. F. (2014). Automatic identi-fication of specieswith neural networks. PeerJ,2, e563. https://doi.org/10.7717/peerj.563
Hodgson,A.,Kelly,N.,&Peel,D.(2013).Unmannedaerialvehicles(UAVs)for surveying Marine Fauna: A dugong case study. PLoS ONE, 8, 1–15.
Howland,J.C.,Macfarlane,N.B.W.,&Tyack,P. (2012).Precisegeopo-sitioningofmarinemammalsusingstereophotogrammetry.OCEANS 2012 MTS/IEEE: Harnessing the Power of the Ocean.pp.1-6.https://doi.org/10.1109/OCEANS.2012.6404859.
Joly,A.,Goëau,H.,Bonnet,P.,Bakić,V.,Barbe,J.,Selmi,S.,…Barthélémy,D. (2014). Interactive plant identification based on social imagedata. Ecological Informatics, 23, 22–34. https://doi.org/10.1016/ j.ecoinf.2013.07.006
Jones, A. M., Cantin, N. E., Berkelmans, R., Sinclair, B., & Negri, A. P.(2008). A 3D modeling method to calculate the surface areas ofcoral branches. Coral Reefs, 27, 521–526. https://doi.org/10.1007/s00338-008-0354-y
Kosmala,M.,Crall,A.,Cheng,R.,Hufkens,K.,Henderson,S.,&Richardson,A. (2016).Seasonspotter:Usingcitizensciencetovalidateandscaleplantphenologyfromnear-surfaceremotesensing.Remote Sensing,8,726.https://doi.org/10.3390/rs8090726
Kühl, H. S., & Burghardt, T. (2013). Animal biometrics: Quantifying anddetectingphenotypicappearance.Trends in Ecology and Evolution,28,432–441.https://doi.org/10.1016/j.tree.2013.02.013
Kvilekval, K., Fedorov, D., Obara, B., Singh, A., & Manjunath, B. S.(2009). Bisque: A platform for bioimage analysis and management.Bioinformatics, 26, 544–552. https://doi.org/10.1093/bioinformatics/btp699
LaRue,M.A., Stapleton, S., &Anderson,M. (2016). Feasibility of usinghigh-resolution satellite imagery to assess vertebrate wildlife popu-lations. Conservation Biology, 31, 213–220. https://doi.org/10.1111/cobi.12809
LaRue,M.A.,Stapleton,S.,Porter,C.,Atkinson,S.,Atwood,T.,Dyck,M.,&Lecomte,N.(2015).Testingmethodsforusinghigh-resolutionsatel-liteimagerytomonitorpolarbearabundanceanddistribution.Wildlife Society Bulletin,39,772–779.https://doi.org/10.1002/wsb.596
Lavy,A.,Eyal,G.,Neal,B.,Keren,R.,Loya,Y.,& Ilan,M. (2015).Aquick,easy and non-intrusive method for underwater volume and sur-face area evaluation of benthic organisms by 3D computer mod-elling. Methods in Ecology and Evolution, 6, 521–531. https://doi.org/10.1111/2041-210X.12331
LeCun,Y.,Bengio,Y.,&Hinton,G.(2015).Deeplearning.Nature,521,436–444.https://doi.org/10.1038/nature14539
Levy, K., Lerner, A., & Shashar, N. (2014). Mate choice and body pat-tern variations in the Crown Butterfly fish Chaetodon paucifascia-tus (Chaetodontidae). Biology Open, 3, 1245–1251. https://doi.org/10.1242/bio.20149175
544 | Journal of Animal Ecology WEINSTEIN
Liu,C.C.,Chen,Y.H.,&Wen,H.L.(2015).Supportingtheannualinterna-tionalblack-faced spoonbill censuswitha low-costunmannedaerialvehicle.Ecological Informatics,30,170–178.https://doi.org/10.1016/j.ecoinf.2015.10.008
Lynch,H.J.,White,R.,Black,A.D.,&Naveen,R. (2012).Detection,dif-ferentiation, and abundance estimation of penguin species by high-resolution satellite imagery. Polar Biology, 35, 963–968. https://doi.org/10.1007/s00300-011-1138-3
Lytle,D.A.,Martínez-Muñoz,G.,Zhang,W.,Larios,N.,Shapiro,L.,Paasch,R., … Dietterich, T. G. (2010).Automated processing and identifica-tion of benthic invertebrate samples. Journal of the North American Benthological Society,29,867–874.https://doi.org/10.1899/09-080.1
Marburg,A.,&Bigham,K.(2016).Deep learning for benthic fauna identifica-tion.OCEANS2016MTS/IEEEMonterey,pp.1–5.
Matuska,S.,Hudec,R.,Kamencay,P.,Benco,M.,&Zachariasova,M.(2014).Classification of wild animals based on SVM and local descriptors.AASRI Procedia,9,25–30.https://doi.org/10.1016/j.aasri.2014.09.006
McDowall, P., & Lynch, H. J. (2017). Ultra-fine scale spatially-integratedmappingofhabitatandoccupancyusingstructure.PLoS ONE,12,1–16.https://doi.org/10.1371/journal.pone.0166773
Mohanty,S.P.,Hughes,D.P.,&Salathé,M.(2016).Usingdeeplearningforimage-basedplantdiseasedetection.Frontiers in Plant Science,7,1–7.https://doi.org/10.3389/fpls.2016.01419
Naumann,M.S.,Niggl,W.,Laforsch,C.,Glaser,C.,&Wild,C.(2009).Coralsurface area quantification-evaluation of established techniques bycomparison with computer tomography. Coral Reefs, 28, 109–117.https://doi.org/10.1007/s00338-008-0459-3
Olsen, A. M., &Westneat, M. W. (2015). StereoMorph: An R packagefor the collection of 3D landmarks and curves using a stereo cam-eraset-up.Methods in Ecology and Evolution,6,351–356.https://doi.org/10.1111/2041-210X.12326
Pennekamp,F.,&Schtickzelle,N. (2013). Implementing imageanalysis inlaboratory-based experimental systems for ecology and evolution:Ahands-onguide.Methods in Ecology and Evolution,4,483–492.https://doi.org/10.1111/2041-210X.12036
Pimm,S.L.,Alibhai,S.,Bergl,R.,Dehgan,A.,Giri,C.,Jewell,Z.,…Loarie,S. (2015). Emerging technologies to conserve biodiversity. Trends in Ecology & Evolution, 30, 685–696. https://doi.org/10.1016/j.tree.2015.08.008
Price Tack, J. L.,West, B. S., McGowan, C. P., Ditchkoff, S. S., Reeves,S. J., Keever, A. C., & Grand, J. B. (2016). AnimalFinder: A semi-automatedsystemforanimaldetectionintime-lapsecameratrapim-ages. Ecological Informatics, 36, 145–151. https://doi.org/10.1016/j.ecoinf.2016.11.003
Qin,H.,Li,X.,Liang,J.,Peng,Y.,&Zhang,C.(2016).DeepFish:Accurateun-derwaterlivefishrecognitionwithadeeparchitecture.Neurocomputing,187,49–58.https://doi.org/10.1016/j.neucom.2015.10.122
Reda,K.,Mateevitsi,V.,&Offord,C.(2013).Ahuman-computercollab-orativeworkflowfortheacquisitionandanalysisofterrestrialinsectmovementinbehavioralfieldstudies.EURASIP Journal on Image and Video Processing, 2013, 48. https://doi.org/10.1186/1687-5281- 2013-48
Ren,X.,Han,T.X.,&He,Z. (2013).Ensemblevideoobjectcut inhighlydynamicscenes.Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,pp.1947–1954.
Robie, A. A., Seagraves, K. M., Egnor, S. E. R., & Branson, K. (2017).Machinevisionmethodsforanalyzingsocialinteractions.The Journal of Experimental Biology,220,25–34.https://doi.org/10.1242/jeb.142281
Seymour, A. C., Dale, J., Hammill, M., Halpin, P. N., & Johnston, D.W.(2017). Automated detection and enumeration of marine wildlifeusingunmannedaircraftsystems(UAS)andthermalimagery.Scientific Reports,7,1–10.https://doi.org/10.1038/srep45127
Sobral,A.,&Vacavant,A.(2014).Acomprehensivereviewofbackgroundsubtraction algorithms evaluated with synthetic and real videos.
Computer Vision and Image Understanding, 122, 4–21. https://doi.org/10.1016/j.cviu.2013.12.005
Stauffer,C.,&Grimson,W.E.L.(1999).Adaptive background mixture mod-els for real-time tracking. Proceedings. 1999 IEEE Computer SocietyConference on ComputerVision and Pattern Recognition, pp. 246–252.IEEEComput.Soc.
Steen, R., & Ski, S. (2014). Video-surveillance system for remote long-terminsituobservations:Recordingdielcavityuseandbehaviourofwild European lobsters (Homarus gammarus).Marine and Freshwater Research,65,1094–1101.https://doi.org/10.1071/MF13139
Stoddard,M.C.,Kilner,R.M.,&Town,C. (2014).Patternrecognitional-gorithm reveals how birds evolve individual egg pattern signatures.Nature Communications,5,4117.
Stoddard,M.C.,Kupán,K.,Eyster,H.N.,Rojas-Abreu,W.,Cruz-López,M.,Serrano-Meneses,M.A.,&Küpper,C.(2016).Camouflageandclutchsurvival inplovers and terns.Scientific Reports,6, 32059.https://doi.org/10.1038/srep32059
Sun,X.,Shi,J.,Dong,J.,&Wang,X.(2017).Fishrecognitionfromlow-res-olution underwater images. In Proceedings of the 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics,CISP-BMEI2016,pp.471–476.
Swanson,A.,Kosmala,M., Lintott,C.,&Packer,C. (2016).Ageneralizedapproachforproducing,quantifying,andvalidatingcitizensciencedatafromwildlife images.Conservation Biology, 30, 520–531. https://doi.org/10.1111/cobi.12695
Swanson,A.,Kosmala,M.,Lintott,C.,Simpson,R.,Smith,A.,&Packer,C.(2015).SnapshotSerengeti,high-frequencyannotatedcameratrapim-agesof40mammalianspeciesinanAfricansavanna.Scientific Data,2,150026.https://doi.org/10.1038/sdata.2015.26
Tankus, A., & Yeshurun, Y. (2009). Computer vision, camouflage break-ingandcountershading.Philosophical Transactions of the Royal Society of London Series B, Biological Sciences, 364, 529–536. https://doi.org/10.1098/rstb.2008.0211
Torney,C.J.,Dobson,A.P.,Borner,F.,Lloyd-Jones,D.J.,Moyer,D.,Maliti,H.T.,…Hopcraft,J.G.C.(2016).Assessingrotation-invariantfeatureclassificationforautomatedwildebeestpopulationcounts.PLoS ONE,11,e0156342.https://doi.org/10.1371/journal.pone.0156342
Town, C., Marshall, A., & Sethasathien, N. (2013). Manta matcher:Automated photographic identification of manta rays using key-point features. Ecology and Evolution, 3, 1902–1914. https://doi.org/10.1002/ece3.587
Troscianko, J., Skelhorn, J., & Stevens, M. (2017). Quantifying camou-flage:Howtopredictdetectabilityfromappearance.BMC Evolutionary Biology,17,7.https://doi.org/10.1186/s12862-016-0854-2
van Andel, A. C., Wich, S. A., Boesch, C., Koh, L. P., Robbins, M. M.,Kelly,J.,&Kuehl,H.S. (2015).Locatingchimpanzeenestsand iden-tifying fruiting trees with an unmanned aerial vehicle. American Journal of Primatology, 77, 1122–1134. https://doi.org/10.1002/ ajp.22446
VanHorn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., …Belongie, S. (2015). Building a bird recognition app and large scaledatasetwith citizen scientists:The fine print in fine-grained datasetcollection. InProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,pp.595–604.
VanHorn,G.,MacAodha,O.,Song,Y.,Shepard,A.,Adam,H.,Perona,P.,&Belongie,S. (2017).The iNaturalistChallenge2017Dataset. arXivpreprintarXiv:1707.06642.
Villon,S.,Chaumont,M.,Subsol,G.,Villéger,S.,Claverie,T.,&Mouillot,D.(2016).Coralreeffishdetectionandrecognitioninunderwatervideosby supervised machine learning: Comparison between deep learn-ing and HOG+SVMmethods. In Blanc-Talon J., Distante C., PhilipsW.,PopescuD.,ScheundersP.(eds)Advanced Concepts for Intelligent Vision Systems. ACIVS 2016.LectureNotes inComputerScience,vol10016.Cham,Switzerland:Springer.
| 545Journal of Animal EcologyWEINSTEIN
Weinstein,B.G.(2015).MotionMeerkat:Integratingmotionvideodetec-tion and ecological monitoring.Methods in Ecology and Evolution, 6,357–362.https://doi.org/10.1111/2041-210X.12320
Weinstein,B.G.(2017).Datafrom:Acomputervisionforanimalecology.Dryad Digital Repository,https://doi.org/10.5061/dryad.b700h
Weinstein,B.G.,&Graham,C.H.(2017).Persistentbillandcorollamatch-ingdespiteshiftingtemporalresourcesintropicalhummingbird-plantinteractions. Ecology Letters, 20, 326–335. https://doi.org/10.1111/ele.12730
Wilber,M.J.,Scheirer,W.J.,Leitner,P.,Heflin,B.,Zott,J.,Reinke,D.,…Boult,T. E. (2013).Animal recognition in theMojaveDesert:Visiontools for field biologists. 2013 IEEE Workshop on Applications of Computer Vision (WACV),pp.206–213.
Witharana,C.,&Lynch,H.J. (2016).Anobject-based imageanalysisap-proach for detecting penguin guano in very high spatial resolutionsatellite images. Remote Sensing, 8, 375. https://doi.org/10.3390/rs8050375
Yang,C.,Wang,L.,Liang,W.,&Møller,A.P.(2016).Docommoncuckoos(Cuculus canorus)possessanoptimal layingbehaviour tomatch theirowneggphenotypetothatoftheirOrientalreedwarbler(Acrocephalus orientalis)hosts?Biological Journal of the Linnean Society,117,422–427.https://doi.org/10.1111/bij.12690
Yang,Z.,Wang,T., Skidmore,A.K.,DeLeeuw,J., Said,M.Y.,&Freer,J.(2014).SpottingEastAfricanmammalsinopensavannahfromspace.PLoS ONE,9,1–16.
Yu, X., Wang, J., Kays, R., Jansen, P. A., Wang, T., & Huang, T. (2013).Automated identification of animal species in camera trap images.EURASIP Journal on Image and Video Processing,2013,52.https://doi.org/10.1186/1687-5281-2013-52
Zeppelzauer, M. (2013). Automated detection of elephants in wildlifevideo. EURASIP Journal on Image and Video Processing,2013,46.https://doi.org/10.1186/1687-5281-2013-46
Zhang,Z.,He,Z.,Cao,G.,&Cao,W.(2016).Animaldetectionfromhighlyclutterednaturalscenesusingspatiotemporalobjectregionproposalsandpatchverification.IEEE Transactions on Multimedia,18,2079–2092.https://doi.org/10.1109/TMM.2016.2594138
Zhang,H.,Korayem,M.,Crandall,D.J.,&Lebuhn,G.(2012).Mining Photo-sharing Websites to Study Ecological Phenomena. International WorldWideWebConferenceCommittee(IW3C2),pp.749–758.
SUPPORTING INFORMATIONAdditional Supporting Information may be found online in thesupportinginformationtabforthisarticle.
How to cite this article:WeinsteinBG.Acomputervisionforanimalecology.J Anim Ecol. 2018;87:533–545. https://doi.org/10.1111/1365-2656.12780
