From Mnemosyne to Terpsichore - the Bilderatlas after the Image LeonardoImpettleonardo.impett@epfl.chÉcolePolytechniqueFédéraledeLausanne,SwitzerlandSabineSusstrunksabine.susstrunk@epfl.chÉcolePolytechniqueFédéraledeLausanne,Switzerland

Introduction ThisstudyconcernsAbyWarburg’s lastandmost

ambitiousproject:theAtlasMnemosyne(orBilderat-las), conceived in August 1926 and truncated threeyearslater,unfinished,byWarburg’ssuddendeathinOctober1929.Mnemosyneconsistsofaseriesoflargeblackpanels,onwhichareattachedavariablenumberof black-and-white photographs of paintings, sculp-tures,tarotcards,stamps,coins,andothertypesofim-ages. The version we use is the one Warburg wasworkingonatthetimeofhisdeath,alsoknownasthe“1-79version”:itincludesaroundathousandimagespinnedon63panels(ThisversionoftheAtlasispub-lishedinvariousprinteditions,andavailableonline).TheBilderatlasisaconceptualmaze-theculminationofalife’sscholarshipinimagesandmemory-throughwhichperhapstheclearestthreadistheconceptofPa-thosformel,orformulafor(theexpressionof)passion.Muchexcellentworkhasbeenwrittenontheconceptbut,tothebestofourknowledge,nobodyhasyettriedto “operationalise” it - to turn it into a sequence ofquantitativeoperations,orinotherwords,intoanal-gorithm(Moretti,2013).

The Pathosformel and its Operationalisation Onthemostbasiclevel,thePathosformeldescribes

theportrayalpassionateemotionsthroughaformula,arepeatablevisualparadigm.ThePathosformelowesmuchofitsforce,asSalvatoreSettishaspointedout,toitscombinationofsemanticopposites:an“oxymo-ronic word, in that it merges in the same term themovementofpathosand the rigidityof the formula-schema”(Settis1997).

Ratherthanattemptingtooperationalisetheentireconceptatonce,wefirstbreakitdownintoitsconstit-uent components: first the morphology of formula,thenthedynamismofpathos.

Weturn to twowell-studiedcasesofWarburgianformula: theDeathofOrpheus(whereWarburg firstnamedtheconceptofPathosformel),andtheNymph,headhuntressandFortuna(Bilderatlaspanels46-48).Looking at Warburg’s examples of the Orpheus-for-mula,reproducedinFigure1,wecanhardlystopour-selves fromspottinga formulawhich repeats acrossthe ages; but how could this intuitive similarity bemeasured?

Ouralgorithmcomesasfollows: 1. Weisolateeachindividualbodyfromits

context. It is clear the Pathosformel re-latestoindividualcharacters-theobjectofstudythusbecomesnotpanelsorpho-tographsintheBilderatlas,butindividualbodies.

2. Wetakeonlytheskeletonsofsuchbodies.Herewe are eliminating colour, clothes,hands, faces, gender, age. This is not tosay that such factors aren’t important:but they are not elementary to the for-mula(seeforexample,AndréJolles’lettertoWarburg23rdDecember1900,wheretheformulaofGhirlandaio’sNymphhopsbetween Judith, Salome, Tobias, Gabrieletc.–seeGhelardi,2004;ortheformulainBilderatlas Panel 47 shared betweenGiambolo-gna’s Samson and Donatello’sJudith.)

3. Wecomparetheseskeletonsbymeasur-ing the angles of the main limbs of thebody,asdescribedinSection5.

Each of these steps is not merely a convenientquantification, but a conceptual wager. This is thestrength of operationalisation as a critical tool: itforcesonetobeexplicitabouttheconceptualchoicesonemakes.

Fig 1: the Death of Orpheus: details from Warburg’s

example figures from classical antiquity to Dürer. Reproduced from Warburg 1905/1999

Anatomy, Emotion and Pose

WarburgwascertainlyinfluencedbyDarwin’sTheExpressionof theEmotions inManandAnimals(Dar-win1872) -which,when talkingabouthumanemo-tions,largelyconcentratesontheface.Indeed,thefirstfigureofthebookisananatomicaldiagramoftheface-“Ishalloftenhavetorefer[...]tothemusclesofthehuman face” (Darwin 1872 p.23).Warburgwas cer-tainly struck by the book - writing in his diary “einBuch,dasmirhilft!”(Gombrich,p.72).Hewasalsoin-terestedandcapableofstudyingthefaceinart(seee.g.hisdiscussionoffacesinGhirlandaio’sConfirmationinSantaTrinita,inTheArtofPortraitureintheFloren-tineBourgeoisie(inWarburg1999p.185),yetneverin relation to Pathos - his descriptions of the Pa-thosformelrelateexclusivelytothebody.

WecanrelateWarburg’sdecisiontothelargepsy-chological literature on emotional recognition frombodies. Psychological studies are based on the LightSpotsModelbyJohansson(1973),oftencalled‘biolog-ical motion’, in which reducing body pose to 10-12points - quite comparable to our own reduction - isjudged to give a ‘compelling impression of humanwalking,running,dancingetc.’.UsingonlyLightSpots,observers can reliably tell gender andemotion fromdynamicpose(Kozlowski1977,Montepare1987).In-deed,ithasbeensuggestedthatouremotionalunder-standingoffacesismoreinfluencedbyourperceptionofthebodythanviceversa(VandenStock2007). Encoding Pathos through Pose

The Atlas is, even by today’s dataset standards,quitesizeable:1000imagesacross63panels,contain-inganorderof103-104depictedhumanfigures.Scala-ble manual annotation is only therefore possiblethrough crowdsourcing, which we did through theCrowdFlowerplatform.

Accuratelyannotatingeveryvisiblefigureinanim-ageisadifficultandambiguoustasks.Additionally,ifdifferent workers annotate different figures in thesameimage,theannotationscannotbecollatedorav-eraged.We thereforedevelopeda two-stageannota-tionprocess:

Humanfiguresareextractedfromthepaintingbydrawingbounding-boxes.Thisisdonethreetimesperimage(bythreeseparateworkers).

Having aggregated the information from the firststage,separate imagesareproducedforeachhumanfigure.Detailedposeinformation(thepositionofma-jorbody-points)isthenaddedbythreeseparatework-ers,withtheinformationaggregated.

Itshouldbeclarifiedthatthedecisiontoannotatebodies in isolation (for annotationaccuracyand just

workercompensation)isquiteseparatetotheearlierconceptual decision to analyse bodies individually,whichrelatestotheobjectofstudy.Itwouldbequitepossibletodoeitheronewithouttheother.

Using this two-stageannotationprocess,wehavepresentlyannotated⅓oftheBilderatlas(bypanels),resulting in1,665aggregatedhumanposes.The col-lection and aggregation of the data are described ingreatertechnicaldetailelsewhere(Impett,2016).

Data Analysis: dimensionality reduction and dimensioned reductionism

Havingencodedourstaticposes,howdoweana-lyseandcompareacollectionofhumanfiguresofdif-ferent sizes, proportions and orientations? We mir-roredtheposeshorizontallyandcontrolledforglobalrotation, ending upwith a 11-dimensional vector P,describingtheanglesofthemainlimbs.

Fromthisangularposevector,wecanusecircular

statisticstofindamorphologicaldistanceDa,bbetweentwoposesPaandPb:

WherePa,iistheithangleofposevectorPa,and||α

istheangularradiandistance:

Thesemorphologicalpose-differencesarepercep-

tually meaningful over short distances. On a largerscale, they become less perceptually significant: is asittingperson‘closer’toalyingorstandingperson?

Inordertomakeourdistanceanalysisperceptuallyrelevant,therefore,wefirstclusteredour1,665posesinto 16 clusters by rotational K-means clustering(Dordet-Berdanet and Wicker 2008). Our two-stageclusteringsystemisthereforeasfollows:

I. K-meansclustering(toproducemeaning-fulclusters)

II. Hierarchical clustering (for within-clus-termorphologicalinformation)

Thenumberof clustersK is chosenby lookingattheinter-clustervarianceoverK.Theresultofthefirststage of clustering is shown in Figure 2; Figure 3showsanexamplesectionfromahierarchicalmapofCluster1.

Fig 2: our 16 pose-clusters

Fig 3: a detail from a dendrogram of different poses within the Bilderatlas, produced by second-stage clustering within

Cluster 2

Unity of the Pathosformeln: from distant to close reading

Someoftheclustersclearlyrepresentphysicalac-tivities -sitting,praying,embracing,dancing-whilstothersseemmoresubtlycommunicativeorexpressiveinnature.Havingreorganisedthe1,665figuresinto16mean-centredclusters,weproceededtotracetheclas-sicalPathosformeln-identifiedintheAtlasbyprimaryandsecondaryliterature-throughourclusters.

ThecanonicalPathosformelnaremainlymytholog-icalfigures(Perseus,Pentheus,Orpheus)orrecurringallegories(Graces,Nymphs,Fortuna).Theywerepre-viouslydescribedasdistinct,andweexpectedtofinda taxonomyof such formulae throughour clusteringanalysis.

On the contrary, the statistical result was muchstronger: a complete morphological unity. AlmosteveryidentifiedPathosformelfallsintoCluster1,withfewfalsepositives-over80%ofthefiguresinCluster1areanidentifiedPathosformel.Thehandfulofexcep-tionsareallborderlinecases,placedinperipheraltoCluster1(Clusters7and13).

Lookingmorecloselyattheimagesthemselves,asinFigure4,thisbecomesvisuallyclear:notonlydothePathosformelnsharecertainposefeatures(most im-portantly,araisedarm)presentnowhereelse in thedataset.Todate,however,theauthorsknowofnoart-historical literaturethathas identifiedsuchmorpho-logicalunity.

Figure 4, clockwise from top-left: Laocoön, Orpheus, Fortuna, Nymph, Judith, Perseus - all except Orpheus are in the Atlas. The identified Pathosformeln share distinguishing features from the other characters in the Atlas: a raised arm,

most often accompanied by a lowered second arm, and a slight twist of the body.

Concluding remarks

Ourmorphologicalmodel forPathosformel issta-tisticallystrong:butwhataretheart-historicalimpli-cations?TheoppositionalsymmetryandraisedarmofCluster1(Fig.2)remindsusofaContrapposto,butthebodiesthemselvesarefarremovedfromsuchclassicalbalance(e.g.Fig.4,top).Ratherthanmovement,ten-sion(betweenupperandlowerbody)seemstobethefundamentalelementofPathosformel-thenatureofwhichwillbethesubjectofasubsequentpublication.

Ourmorphological analysishas shown that staticpose can identifyPathosformeln, and thata studyofstaticposethroughalargecollectionofartisticworkscanidentifylinksacrossstyles,periodsandcultures.

Automaticdetectionofposeisafocusofthecurrentresearch,andwillallowustoexpandourart-historicalmodelsbeyondtheBilderatlas.Humans(thusbodies,andposes)areunsurprisinglythemostcommonfea-tureof humanart, and thereforemake excellent ob-jectsof study foranarthistoryof theLongueDurée(Robb2015).

Bibliography

Darwin,C.(1872).Theexpressionoftheemotionsinmanandanimals.London,JohnMurray

Dortet-Bernadet,J.L.andWicker,N.,(2008).Model-based

clusteringontheunitspherewithanillustrationusinggeneexpressionprofiles. InBiostatistics,9(1),pp.66-80.

Gombrich,E.H.(1970).AbyWarburg,anintellectualbiog-

raphy.ChicagoUniversityPress.Impett, L. and Süsstrunk, S., (2016). Pose and Pa-

thosformelinAbyWarburg’sBilderatlas.InEuropeanConference on Computer Vision (pp. 888-902).SpringerInternationalPublishing.

Johansson,G., (1973).Visualperceptionofbiologicalmo-

tionandamodelforitsanalysis.InPerception&psy-chophysics,14(2),pp.201-211.

Kozlowski,L.T.andCutting,J.E.,(1977).Recognizingthe

sexofawalkerfromadynamicpoint-lightdisplay.InPerception&Psychophysics,21(6),pp.575-580.

Montepare,J.M.,Goldstein,S.B.andClausen,A.,(1987).

The identificationof emotions fromgait information.InJournalofNonverbalBehavior,11(1),pp.33-42.

Moretti,F.,(2013).“Operationalizing”;or,TheFunctionof

Measurement in Modern Literary. Stanford LiteraryLab Pamphlet Series. litlab.stanford. edu/Liter-aryLabPamphlet6

Robb,J.andHarris,O.J.T.(2013).“Bodyworldsandtheir

history:someworkingconcepts.”TheBodyinHistory:EuropefromthePalaeolithictothefuture:7(31).

Settis,“PathosundEthos,MorphologieundFunktion”,inW.

Kemp, G. Mattenklott, M. Wagner, M. Warnke, eds,VorträgeausdemWarburg-Haus,Band1,DeGruyter,Berlin1997,pp.39-44.

VandenStock, J.,Righart,R.andDeGelder,B., (2007).

Bodyexpressionsinfluencerecognitionofemotionsinthefaceandvoice.InEmotion,7(3),p.487.

Warburg,A., trans.D.Britt.(1999)TheRenewalofPaganAntiquity.LosAngeles,GettyResearchInstitute.