Indiscapes Instance Segmentation Networks forLayout Parsing of Historical Indic Manuscripts

Abhishek Prusty Sowmya Aitha Abhishek Trivedi Ravi Kiran SarvadevabhatlaCentre for Visual Information Technology (CVIT)

International Institute of Information Technology Hyderabad (IIIT-H)Gachibowli Hyderabad 500032 INDIA

abhishekprustystudentssowmyaaitharesearchabhishektrivediresearchravikiraniiitacin

AbstractmdashHistorical palm-leaf manuscript and early paperdocuments from Indian subcontinent form an important part ofthe worldrsquos literary and cultural heritage Despite their impor-tance large-scale annotated Indic manuscript image datasets donot exist To address this deficiency we introduce Indiscapesthe first ever dataset with multi-regional layout annotationsfor historical Indic manuscripts To address the challenge oflarge diversity in scripts and presence of dense irregular lay-out elements (eg text lines pictures multiple documents perimage) we adapt a Fully Convolutional Deep Neural Networkarchitecture for fully automatic instance-level spatial layoutparsing of manuscript images We demonstrate the effectivenessof proposed architecture on images from the Indiscapes datasetFor annotation flexibility and keeping the non-technical natureof domain experts in mind we also contribute a custom web-based GUI annotation tool and a dashboard-style analytics portalOverall our contributions set the stage for enabling downstreamapplications such as OCR and word-spotting in historical Indicmanuscripts at scale

Keywords-Document Layout Parsing Palm-leaf manuscripts Semantic Instance Segmentation Deep Neural Networks Indic

I INTRODUCTION

The collection and analysis of historical document imagesis a key component in the preservation of culture and heritageGiven its importance a number of active research effortsexist across the world [1]ndash[6] In this paper we focus onpalm-leaf and early paper documents from the Indian sub-continent In contrast with modern or recent era documentssuch manuscripts are considerably more fragile prone todegradation from elements of nature and tend to have a shortshelf life [7]ndash[9] More worryingly the domain experts whocan decipher such content are small in number and dwindlingTherefore it is essential to access the content within thesedocuments before it is lost forever

Surprisingly no large-scale annotated Indic manuscript im-age datasets exist for the benefit of researchers in the commu-nity In this paper we take a significant step to address thisgap by creating such a dataset Given the large diversity inlanguage script and non-textual regional elements in thesemanuscripts spatial layout parsing is crucial in enablingdownstream applications such as OCR word-spotting style-and-content based retrieval and clustering For this reason wefirst tackle the problem of creating a diverse annotated spatiallayout dataset This has the immediate advantage of bypassing

the hurdle of language and script familiarity for annotatorssince layout annotation does not require any special expertiseunlike text annotation

In general manuscripts from Indian subcontinent pose manyunique challenges (Figure 1) To begin with the documentsexhibit a large multiplicity of languages This is further magni-fied by variations in intra-language script systems Along withtext manuscripts may contain pictures tables non-pictorialdecorative elements in non-standard layouts A unique aspectof Indic and South-East Asian manuscripts is the frequentpresence of holes punched in the document for the purposeof binding [7] [9] [10] These holes cause unnatural gapswithin text lines The physical dimensions of the manuscriptsare typically smaller compared to other historical documentsresulting in a dense content layout Sometimes multiplemanuscript pages are present in a single image Moreoverimaging-related factors such as varying scan quality play a roleas well Given all of these challenges it is important to developrobust and scalable approaches for the problem of layoutparsing In addition given the typical non-technical nature ofdomain experts who study manuscripts it is also important todevelop easy-to-use graphical interfaces for annotation post-annotation visualization and analytics

We make the following contributionsbull We introduce Indiscapes the first ever historical Indic

manuscript dataset with detailed spatial layout annota-tions (Section III)

bull We adapt a deep neural network architecture for instance-level spatial layout parsing of historical manuscript im-ages (Section IV-A)

bull We also introduce a lightweight web-based GUI forannotation and dashboard-style analytics keeping in mindthe non-technical domain experts and the unique layout-level challenges of Indic manuscripts (Section III-B)

II RELATED WORK

A number of research groups have invested significantefforts in the creation and maintenance of annotated pub-licly available historical manuscript image datasets [1]ndash[4][11]ndash[13] Other collections contain character-level and word-level spatial annotations for South-East Asian palm-leafmanuscripts [5] [10] [14] In these latter set of works

arX

iv1

912

0702

5v1

[cs

CV

] 1

5 D

ec 2

019

Fig 1 The five images on the left enclosed by pink dotted line are from the BHOOMI palm leaf manuscript collectionwhile the remaining images (enclosed by blue dotted line) are from the rsquoPenn-in-Handrsquo collection (refer to Section III) Notethe inter-collection differences closely spaced and unevenly written text lines presence of various non-textual layout regions(pictures holes library stamps) physical degradation and presence of multiple manuscripts per image All of these factorspose great challenges for annotation and machine-based parsing

annotations for lines are obtained by considering the polygonalregion formed by union of character bounding boxes as a lineWhile studies on Indic palm-leaf and paper-based manuscriptsexist these are typically conducted on small and often privatecollections of documents [15]ndash[21] No publicly availablelarge-scale annotated dataset of historical Indic manuscriptsexists to the best of our knowledge In contrast with existingcollections our proposed dataset contains a much larger diver-sity in terms of document type (palm-leaf and early paper)scripts and annotated layout elements (see Tables IIII) Anadditional level of complexity arises from the presence ofmultiple manuscript pages within a single image (see Fig 1)

A number of contributions can also be found for the task ofhistorical document layout parsing [22]ndash[25] Wei et al [23]explore the effect of using a hybrid feature selection methodwhile using autoencoders for semantic segmentation in fivehistorical English and Medieval European manuscript datasetsChen et al [25] explore the use of Fully Convolutional Net-works (FCN) for the same datasets Barakat et al [26] proposea FCN for segmenting closely spaced arbitrarily orientedtext lines from an Arabic manuscript dataset The mentionedapproaches coupled with efforts to conduct competitions onvarious aspects of historical document layout analysis haveaided progress in this area [27]ndash[29] A variety of layoutparsing approaches including those employing the modernparadigm of deep learning have been proposed for Indic [18][20] [21] [30] and South-East Asian [14] [24] [31]ndash[33]palm-leaf and paper manuscript images However existingapproaches typically employ brittle hand-crafted features ordemonstrate performance on datasets which are limited interms of layout diversity Similar to many recent workswe employ Fully Convolutional Networks in our approachHowever a crucial distinction lies in our formulation of layoutparsing as an instance segmentation problem rather than just

a semantic segmentation problem This avoids the problem ofclosely spaced layout regions (eg lines) being perceived ascontiguous blobs

The ready availability of annotation and analysis tools hasfacilitated progress in creation and analysis of historical docu-ment manuscripts [34]ndash[36] The tool we propose in the papercontains many of the features found in existing annotationsystems However some of these systems are primarily ori-ented towards single-user offline annotation and do not enablea unified management of annotation process and monitoringof annotator performance In contrast our web-based systemaddresses these aspects and provides additional capabilitiesMany of the additional features in our system are tailored forannotation and examining annotation analytics for documentswith dense and irregular layout elements especially thosefound in Indic manuscripts In this respect our annotationsystem is closer to the recent trend of collaborative cloudweb-based annotation systems and services [37]ndash[39]

III INDISCAPES THE INDIC MANUSCRIPT DATASET

The Indic manuscript document images in our dataset areobtained from two sources The first source is the pub-licly available Indic manuscript collection from Universityof Pennsylvaniarsquos Rare Book and Manuscript Library [40]also referred to as Penn-in-Hand (PIH) From the 2880 Indicmanuscript book-sets1 we carefully curated 193 manuscriptimages for annotation Our curated selection aims to maximizethe diversity of the dataset in terms of various attributes such asthe extent of document degradation script language presenceof non-textual elements (eg pictures tables) and number oflines Some images contain multiple manuscript pages stackedvertically or horizontally (see bottom-left image in Figure 1)The second source for manuscript images in our dataset is

1A book-set is a sequence of manuscript images

Character Line Segment Character Component Hole Page Boundary Library Marker Decorator Picture Physical Degradation Boundary Line(CLS) (CC) (H) (PB) (LM) (D) (P) (PD) (BL)

PIH 2401 494 minus 256 32 59 94 34 395BHOOMI 2440 210 565 316 133 minus minus 2078 minus

Combined 4841 704 565 572 165 59 94 2112 395

TABLE I Counts for various annotated region types in INDISCAPES dataset The abbreviations used for region types are givenbelow each region type

Train Validation Test Total

PIH 116 28 49 193BHOOMI 236 59 20 315

Total 352 87 69 508

TABLE II Dataset splits used for learning and inference

Script Source Document Count

Devanagari PIH 193Nandinagari BHOOMI 2

Telugu BHOOMI 75Grantha BHOOMI 238

TABLE III Scripts in the INDISCAPES dataset

BHOOMI an assorted collection of 315 images sourced frommultiple Oriental Research Institutes and libraries across IndiaAs with the first collection we chose a subset intended tomaximize the overall diversity of the dataset However thislatter set of images are characterized by a relatively inferiordocument quality presence of multiple languages and from alayout point of view predominantly contain long closely andirregularly spaced text lines binding holes and degradations(Figure 1) Though some document images contain multiplemanuscripts we do not attempt to split the image into mul-tiple pages While this poses a challenge for annotation andautomatic image parsing retaining such images in the dataseteliminates manualsemi-automatic intervention As our resultsshow our approach can successfully handle such multi-pagedocuments thereby making it truly an end-to-end system

Overall our dataset contains 508 annotated Indicmanuscripts Some salient aspects of the dataset can beviewed in Table I and a pictorial illustration of layout regionscan be viewed in Figure 4 Note that multiple regions canoverlap unlike existing historical document datasets whichtypically contain disjoint region annotations

For the rest of the section we discuss the challengesassociated with annotating Indic manuscripts (Section III-A)and our web-based annotation tool (Section III-B)

A Annotation Challenges

A variety of unique challenges exist in the context ofannotating Indic manuscript layouts The challenges arise fromthree major sources

Content The documents are written in a large variety ofIndic languages Some languages even exhibit intra-languagescript variations A large pool of annotators familiar with thelanguages and scripts present in the corpus is required toensure proper annotation of lines and character componentsLayout Unlike some of the existing datasets Indicmanuscripts contain non-textual elements such as color pic-tures tables and document decorations These elements arefrequently interspersed with text in non-standard layouts Inmany cases the manuscripts contain one or more physicalholes designed for a thread-like material to pass throughand bind the leaves together as a book Such holes vary interms of spatial location count and hole diameter When theholes are present in the middle of the document they causea break in the contiguity of lines In some documents theline contiguity is broken by a lsquovirtualrsquo hole-like gap possiblyintended for creation of the punched hole at a future timeIn many cases the separation between lines is extremelysmall The handwritten nature of these documents and thesurface material result in extremely uneven lines necessitatingmeticulous and slow annotation If multiple manuscript pagesare present the stacking order could be horizontal or verticalOverall the sheer variety in layout elements poses a significantchallenge not only for annotation but also for automatedlayout parsingDegradations Historical Indic manuscripts tend to be inher-ently fragile and prone to damage due to various sourcesndash wood-and-leaf-boring insects humidity seepage improperstorage and handling etc While some degradations cause theedges of the document to become frayed others manifest asirregularly shaped perforations in the document interior It maybe important to identify such degradations before attemptinglexically-focused tasks such as OCR or word-spotting

B Annotation Tool

Keeping the aforementioned challenges in mind we in-troduce a new browser-based annotation tool (see Figure2) The tool is designed to operate both stand-alone andas a web-service The web-service mode enables featuressuch as distributed parallel sessions by registered annotatorsdashboard-based live session monitoring and a wide varietyof annotation-related analytics On the front-end a freehandregion option is provided alongside the usual rectangle andpolygon to enable maximum annotation flexibility The web-service version also features a lsquoCorrection-modersquo which en-ables annotators to correct existing annotations from previousannotators Additionally the tool has been designed to enable

Fig 2 Screenshots of our web-based annotator (left) and analytics dashboard (right)

lexical (text) annotations in future

IV INDIC MANUSCRIPT LAYOUT PARSING

To succeed at layout parsing of manuscripts we requirea system which can accurately localize various types ofregions (eg text lines isolated character components physicaldegradation pictures holes) More importantly we require asystem which can isolate individual instances of each region(eg multiple text lines) in the manuscript image Also in ourcase the annotation regions for manuscripts are not disjointand can overlap (eg The annotation region for a text line canoverlap with the annotation region of a hole (see Figure 4))Therefore we require a system which can accommodate suchoverlaps To meet all of these requirements we model ourproblem as one of semantic instance-level segmentation andemploy the Mask R-CNN [41] architecture which has provento be very effective at the task of object-instance segmenta-tion in photos Next we briefly describe the Mask R-CNNarchitecture and our modifications of the same Subsequentlywe provide details related to implementation (Section IV-B)model training (Section IV-B1) and inference (Section IV-B2)

A Network Architecture

The Mask-RCNN architecture contains three stages as de-scribed below (see Figure 3)Backbone The first stage referred to as the backbone isused to extract features from the input image It consistsof a convolutional network combined with a feature-pyramidnetwork [42] thereby enabling multi-scale features to beextracted We use the first four blocks of ResNet-50 [43] asthe convolutional networkRegion Proposal Network (RPN) This is a convolutionalnetwork which scans the pyramid feature map generatedby the backbone network and generates rectangular regionscommonly called lsquoobject proposalsrsquo which are likely to containobjects of interest For each level of the feature pyramid andfor each spatial location at a given level a set of level-specificbounding boxes called anchors are generated The anchorstypically span a range of aspect ratios (eg 1 2 1 1 2 1)for flexibility in detection For each anchor the RPN networkpredicts (i) the probability of an object being present (lsquoobject-ness scorersquo) (ii) offset coordinates of a bounding box relative to

location of the anchor The generated bounding boxes are firstfiltered according to the lsquoobjectness scorersquo From boxes whichsurvive the filtering those that overlap with the underlyingobject above a certain threshold are chosen After applyingnon-maximal suppression to remove overlapping boxes withrelatively smaller objectness scores the final set of boxeswhich remain are termed lsquoobject proposalsrsquo or Regions-of-Interest (RoI)Multi-Task Branch Networks The RoIs obtained from RPNare warped into fixed dimensions and overlaid on feature mapsextracted from the backbone to obtain RoI-specific featuresThese features are fed to three parallel task sub-networks Thefirst sub-network maps these features to region labels (egHoleCharacter-Line-Segment) while the second sub-network maps the RoI features to bounding boxes The thirdsub-network is fully convolutional and maps the features tothe pixel mask of the underlying region Note that the abilityof the architecture to predict masks independently for eachRoI plays a crucial role in obtaining instance segmentationsAnother advantage is that it naturally addresses situationswhere annotations or predictions overlap

B Implementation Details

The dataset splits used for training validation and testphases can be seen in Table II All manuscript images areadaptively resized to ensure the width does not exceed 1024pixels The images are padded with zeros such that the inputto the deep network has spatial dimensions of 1024 times 1024The ground truth region masks are initially subjected to asimilar resizing procedure Subsequently they are downsizedto 28 times 28 in order to match output dimensions of the masksub-network

1) Training The network is initialized with weights ob-tained from a Mask R-CNN trained on the MS-COCO [44]dataset with a ResNet-50 backbone We found that this re-sults in faster convergence and stabler training compared tousing weights from a Mask-RCNN trained on ImageNet [45]or training from scratch Within the RPN network we usecustom-designed anchors of 5 different scales and with 3different aspect ratios Specifically we use the followingaspect ratios ndash 1113110 ndash keeping in mind the typicalspatial extents of the various region classes We also limit

Predicted region types

ROI Align

FC layer

FC layer

FC layer

Input Image

Feature Pyramid

Line

Region Proposals

Detected Bounding

boxesRPN

CLS

PB

P

Region Classifier

Bounding box regressor

Instance Mask Predictor

ResNet-50 Backbone

Page Boundary

Picture

Created by Unlicensed Version

Created by Unlicensed Version

Created by Unlicensed Version

Fig 3 The architecture adopted for Indic Manuscript Layout Parsing Refer to Section IV for details

Average IoU Average Per pixel AccuracyDataset darr H CLS PD PB CC P D LM BL

PIH minus 74179257 minus 86909637 52847485 60498221 523617 50295697 29454314BHOOMI 79299995 29074367 8721298 91099922 32504719 minus minus 38254998 minusCombined 79299995 57777479 8721298 88479744 45876537 60498221 523617 42935270 29454314

TABLE IV Class-wise average IoUs and per-pixel accuracies on the test set Refer to Table I for full names of abbreviatedregion types listed at top of the table

AP50 AP75 AP

PIH 7978 6011 4964Bhoomi 3688 1495 1800

Combined 6476 4430 3857

TABLE V AP at IoU thresholds 50 75 and overall APaveraged over IoU range for test set

the number of RoIs (lsquoobject proposalsrsquo) to 512 We usecategorical cross entropy loss LRPN for RPN classificationnetwork Within the task branches we use categorical crossentropy loss Lr for region classification branch smooth L1loss [46] (Lbb) for final bounding box prediction and per-pixel binary cross entropy loss Lmask for mask predictionThe total loss is a convex combination of these losses ieL = λRPNLRPN + λrLr + λbbLbb + λmaskLmask Theweighting factors (λs) are set to 1 However to ensure priorityfor our task of interest namely mask prediction we setλmask = 2 For optimization we use Stochastic GradientDescent (SGD) optimizer with a gradient norm clipping valueof 05 The batch size momentum and weight decay are setto 1 09 and 10minus3 respectively Given the relatively smallersize of our manuscript dataset compared to the photo dataset

(MS-COCO) used to originally train the base Mask R-CNNwe adopt a multi-stage training strategy For the first stage (30epochs) we train only the task branch sub-networks using alearning rate of 10minus3 while freezing weights in the rest of theoverall network This ensures that the task branches are fine-tuned for the types of regions contained in manuscript imagesFor the second stage (20 epochs) we additionally train stage-4 and up of the backbone ResNet-50 This enables extractionof appropriate semantic features from manuscript images Theomission of the initial 3 stages in the backbone for trainingis due to the fact that they provide generic re-usable low-level features To ensure priority coverage of hard-to-localizeregions we use focal loss [47] for mask generation For thefinal stage (15 epochs) we train the entire network using alearning rate of 10minus4

2) Inference During inference the images are rescaledand processed using the procedure described at the beginningof the subsection The number of RoIs retained after non-maximal suppression (NMS) from the RPN is set to 1000From these we choose the top 100 region detections withobjectness score exceeding 05 and feed the correspondingRoIs to the mask branch sub-network for mask generationIt is important to note that this strategy is different from theparallel generation of outputs and use of the task sub-networksduring training The generated masks are then binarized usingan empirically chosen threshold of 04 and rescaled to their

Fig 4 Ground truth annotations (left) and predicted instance segmentations (right) for test set images Note that we use coloredshading only to visualize individual region instances and not to color-code region types The region label abbreviations areshown alongside the regions CLS Character Line Segment PB Page Boundary H Hole BL Boundary Line CC Character Component PD Physical Degradation

original size using bilinear interpolation On these generatedmasks NMS with a threshold value of 05 is applied to obtainthe final set of predicted masks

C Evaluation

For quantitative evaluation we compute Average Precision(AP) for a particular IoU threshold a measure widely reportedin instance segmentation literature [44] [48] We specificallyreport AP50 and AP75 corresponding to AP at IoU thresholds50 and 75 respectively [41] In addition we report an overallscore by averaging AP at different IoU thresholds ranging from05 to 095 in steps of 005

The AP measure characterizes performance at documentlevel To characterize performance for each region type wereport two additional measures [25] ndash average class-wise IoU(cwIoU) and average class-wise per-pixel accuracy (cwAcc)Consider a fixed test document k Suppose there are ri regionsof class i and let IoUr denote the IoU score for one suchregion r ie 1 6 r 6 ri The per-class IoU score for class iand document k is computed as cwIoUd

i =sum

r IoUr

ri Suppose

there are Ni documents containing at least a single region ofclass i in ground-truth The overall per-class IoU score forclass i is computed as cwIoU i =

sumd cwIoUd

i

Ni In a similar

manner we define class-wise pixel accuracy pwAccdi at docu-ment level and average it across all the documents containingclass i ie cwAcci =

sumd pwAccdi

Ni Note that our approach

for computing class-wise scores prevents documents with arelatively larger number of class instances from dominating thescore and in this sense differs from existing approaches [25]

V RESULTS

We report quantitative results using the measures describedin Section IV-C Table IV reports Average Precision and TableV reports class-wise average IOUs and per-pixel accuraciesQualitative results can be viewed in Figure 4 Despite thechallenges posed by manuscripts our model performs rea-sonably well across a variety of classes As the qualitativeresults indicate the model predicts accurate masks for almostall the regions The results also indicate that our model handlesoverlap between Holes and Character line segments wellFrom ablative experiments we found that our choice of focalloss was crucial in obtaining accurate mask boundaries Unliketraditional semantic segmentation which would have produceda single blob-like region for line segments our instance-basedapproach isolates each text line separately Additionally theclear demarcation between Page-Boundary and background in-dicates that our system identifies semantically relevant regionsfor downstream analysis As the result at the bottom of Figure4 shows our system can even handle images with multiplepages thus removing the need for any pre-processing relatedto isolation of individual pages

From quantitative results we observe that Holes Characterline segments Page boundary and Pictures are parsed thebest while Physical degradations are difficult to parse dueto the relatively small footprint and inconsistent patterns indegradations The results show that performance for Penn

in Hand (PIH) documents is better compared to BHOOMImanuscripts We conjecture that the presence of closely spacedand unevenly written lines in latter is the cause In ourapproach two (or more) objects may share the same boundingbox in terms of overlap and it is not possible to determinewhich box to choose during mask prediction Consequentlyan underlying linersquos boundary may either end up not beingdetected or the predicted mask might be poorly localizedHowever this is not a systemic problem since our modelachieves good performance even for very dense BHOOMIdocument line layouts

VI CONCLUSION

Via this paper we propose Indiscapes the first dataset withlayout annotations for historical Indic manuscripts We believethat the availability of layout annotations will play a crucialrole in reducing the overall complexity for OCR and othertasks such as word-spotting style-and-content based retrievalIn the long-term we intend to expand the dataset not onlynumerically but also in terms of layout script and languagediversity As a significant contribution we have also adapted adeep-network based instance segmentation framework custommodified for fully automatic layout parsing Given the generalnature of our framework advances in instance segmentationapproaches can be leveraged thereby improving performanceover time Our proposed web-based annotator system althoughdesigned for Indic manuscripts is flexible and could be reusedfor similar manuscripts from Asian subcontinent We intend toexpand the capabilities of our annotator system in many usefulways For instance the layout estimated by our deep-networkcould be provided to annotators for correction thus reduc-ing annotation efforts Finally we plan to have our datasetinstance segmentation system and annotator system publiclyavailable This would enable large-scale data collection andautomated analysis efforts for Indic as well as other historicalAsian manuscripts The repositories related to the systemspresented in this paper and the Indiscapes dataset can beaccessed at httpsihdiaiiitacin

ACKNOWLEDGMENT

We would like to thank Dr Sai Susarla for enabling accessto the Bhoomi document collection We also thank PoreddyMourya Kumar Reddy Gollapudi Sai Vamsi Krishna for theircontributions related to dashboard and various annotators fortheir labelling efforts

REFERENCES

[1] C Reul M Dittrich and M Gruner ldquoCase study of a highly automatedlayout analysis and ocr of an incunabulumrsquoder heiligen lebenrsquo(1488)rdquoin Proc 2nd Intl Conf on Digital Access to Textual Cultural HeritageACM 2017 pp 155ndash160 1

[2] U Springmann and A Luedeling ldquoOcr of historical printings with anapplication to building diachronic corpora A case study using the ridgesherbal corpusrdquo Digital Humanities Quarterly no 2 2017 1

[3] F Simistira M Seuret N Eichenberger A Garz M Liwicki and R In-gold ldquoDiva-hisdb A precisely annotated large dataset of challengingmedieval manuscriptsrdquo in ICFHR IEEE 2016 pp 471ndash476 1

Fig 1 The five images on the left enclosed by pink dotted line are from the BHOOMI palm leaf manuscript collectionwhile the remaining images (enclosed by blue dotted line) are from the rsquoPenn-in-Handrsquo collection (refer to Section III) Notethe inter-collection differences closely spaced and unevenly written text lines presence of various non-textual layout regions(pictures holes library stamps) physical degradation and presence of multiple manuscripts per image All of these factorspose great challenges for annotation and machine-based parsing

annotations for lines are obtained by considering the polygonalregion formed by union of character bounding boxes as a lineWhile studies on Indic palm-leaf and paper-based manuscriptsexist these are typically conducted on small and often privatecollections of documents [15]ndash[21] No publicly availablelarge-scale annotated dataset of historical Indic manuscriptsexists to the best of our knowledge In contrast with existingcollections our proposed dataset contains a much larger diver-sity in terms of document type (palm-leaf and early paper)scripts and annotated layout elements (see Tables IIII) Anadditional level of complexity arises from the presence ofmultiple manuscript pages within a single image (see Fig 1)

A number of contributions can also be found for the task ofhistorical document layout parsing [22]ndash[25] Wei et al [23]explore the effect of using a hybrid feature selection methodwhile using autoencoders for semantic segmentation in fivehistorical English and Medieval European manuscript datasetsChen et al [25] explore the use of Fully Convolutional Net-works (FCN) for the same datasets Barakat et al [26] proposea FCN for segmenting closely spaced arbitrarily orientedtext lines from an Arabic manuscript dataset The mentionedapproaches coupled with efforts to conduct competitions onvarious aspects of historical document layout analysis haveaided progress in this area [27]ndash[29] A variety of layoutparsing approaches including those employing the modernparadigm of deep learning have been proposed for Indic [18][20] [21] [30] and South-East Asian [14] [24] [31]ndash[33]palm-leaf and paper manuscript images However existingapproaches typically employ brittle hand-crafted features ordemonstrate performance on datasets which are limited interms of layout diversity Similar to many recent workswe employ Fully Convolutional Networks in our approachHowever a crucial distinction lies in our formulation of layoutparsing as an instance segmentation problem rather than just

a semantic segmentation problem This avoids the problem ofclosely spaced layout regions (eg lines) being perceived ascontiguous blobs

The ready availability of annotation and analysis tools hasfacilitated progress in creation and analysis of historical docu-ment manuscripts [34]ndash[36] The tool we propose in the papercontains many of the features found in existing annotationsystems However some of these systems are primarily ori-ented towards single-user offline annotation and do not enablea unified management of annotation process and monitoringof annotator performance In contrast our web-based systemaddresses these aspects and provides additional capabilitiesMany of the additional features in our system are tailored forannotation and examining annotation analytics for documentswith dense and irregular layout elements especially thosefound in Indic manuscripts In this respect our annotationsystem is closer to the recent trend of collaborative cloudweb-based annotation systems and services [37]ndash[39]

III INDISCAPES THE INDIC MANUSCRIPT DATASET

The Indic manuscript document images in our dataset areobtained from two sources The first source is the pub-licly available Indic manuscript collection from Universityof Pennsylvaniarsquos Rare Book and Manuscript Library [40]also referred to as Penn-in-Hand (PIH) From the 2880 Indicmanuscript book-sets1 we carefully curated 193 manuscriptimages for annotation Our curated selection aims to maximizethe diversity of the dataset in terms of various attributes such asthe extent of document degradation script language presenceof non-textual elements (eg pictures tables) and number oflines Some images contain multiple manuscript pages stackedvertically or horizontally (see bottom-left image in Figure 1)The second source for manuscript images in our dataset is

1A book-set is a sequence of manuscript images

Character Line Segment Character Component Hole Page Boundary Library Marker Decorator Picture Physical Degradation Boundary Line(CLS) (CC) (H) (PB) (LM) (D) (P) (PD) (BL)

PIH 2401 494 minus 256 32 59 94 34 395BHOOMI 2440 210 565 316 133 minus minus 2078 minus

Combined 4841 704 565 572 165 59 94 2112 395

TABLE I Counts for various annotated region types in INDISCAPES dataset The abbreviations used for region types are givenbelow each region type

Train Validation Test Total

PIH 116 28 49 193BHOOMI 236 59 20 315

Total 352 87 69 508

TABLE II Dataset splits used for learning and inference

Script Source Document Count

Devanagari PIH 193Nandinagari BHOOMI 2

Telugu BHOOMI 75Grantha BHOOMI 238

TABLE III Scripts in the INDISCAPES dataset

BHOOMI an assorted collection of 315 images sourced frommultiple Oriental Research Institutes and libraries across IndiaAs with the first collection we chose a subset intended tomaximize the overall diversity of the dataset However thislatter set of images are characterized by a relatively inferiordocument quality presence of multiple languages and from alayout point of view predominantly contain long closely andirregularly spaced text lines binding holes and degradations(Figure 1) Though some document images contain multiplemanuscripts we do not attempt to split the image into mul-tiple pages While this poses a challenge for annotation andautomatic image parsing retaining such images in the dataseteliminates manualsemi-automatic intervention As our resultsshow our approach can successfully handle such multi-pagedocuments thereby making it truly an end-to-end system

Overall our dataset contains 508 annotated Indicmanuscripts Some salient aspects of the dataset can beviewed in Table I and a pictorial illustration of layout regionscan be viewed in Figure 4 Note that multiple regions canoverlap unlike existing historical document datasets whichtypically contain disjoint region annotations

For the rest of the section we discuss the challengesassociated with annotating Indic manuscripts (Section III-A)and our web-based annotation tool (Section III-B)

A Annotation Challenges

A variety of unique challenges exist in the context ofannotating Indic manuscript layouts The challenges arise fromthree major sources

Content The documents are written in a large variety ofIndic languages Some languages even exhibit intra-languagescript variations A large pool of annotators familiar with thelanguages and scripts present in the corpus is required toensure proper annotation of lines and character componentsLayout Unlike some of the existing datasets Indicmanuscripts contain non-textual elements such as color pic-tures tables and document decorations These elements arefrequently interspersed with text in non-standard layouts Inmany cases the manuscripts contain one or more physicalholes designed for a thread-like material to pass throughand bind the leaves together as a book Such holes vary interms of spatial location count and hole diameter When theholes are present in the middle of the document they causea break in the contiguity of lines In some documents theline contiguity is broken by a lsquovirtualrsquo hole-like gap possiblyintended for creation of the punched hole at a future timeIn many cases the separation between lines is extremelysmall The handwritten nature of these documents and thesurface material result in extremely uneven lines necessitatingmeticulous and slow annotation If multiple manuscript pagesare present the stacking order could be horizontal or verticalOverall the sheer variety in layout elements poses a significantchallenge not only for annotation but also for automatedlayout parsingDegradations Historical Indic manuscripts tend to be inher-ently fragile and prone to damage due to various sourcesndash wood-and-leaf-boring insects humidity seepage improperstorage and handling etc While some degradations cause theedges of the document to become frayed others manifest asirregularly shaped perforations in the document interior It maybe important to identify such degradations before attemptinglexically-focused tasks such as OCR or word-spotting

B Annotation Tool

Keeping the aforementioned challenges in mind we in-troduce a new browser-based annotation tool (see Figure2) The tool is designed to operate both stand-alone andas a web-service The web-service mode enables featuressuch as distributed parallel sessions by registered annotatorsdashboard-based live session monitoring and a wide varietyof annotation-related analytics On the front-end a freehandregion option is provided alongside the usual rectangle andpolygon to enable maximum annotation flexibility The web-service version also features a lsquoCorrection-modersquo which en-ables annotators to correct existing annotations from previousannotators Additionally the tool has been designed to enable

Fig 2 Screenshots of our web-based annotator (left) and analytics dashboard (right)

lexical (text) annotations in future

IV INDIC MANUSCRIPT LAYOUT PARSING

To succeed at layout parsing of manuscripts we requirea system which can accurately localize various types ofregions (eg text lines isolated character components physicaldegradation pictures holes) More importantly we require asystem which can isolate individual instances of each region(eg multiple text lines) in the manuscript image Also in ourcase the annotation regions for manuscripts are not disjointand can overlap (eg The annotation region for a text line canoverlap with the annotation region of a hole (see Figure 4))Therefore we require a system which can accommodate suchoverlaps To meet all of these requirements we model ourproblem as one of semantic instance-level segmentation andemploy the Mask R-CNN [41] architecture which has provento be very effective at the task of object-instance segmenta-tion in photos Next we briefly describe the Mask R-CNNarchitecture and our modifications of the same Subsequentlywe provide details related to implementation (Section IV-B)model training (Section IV-B1) and inference (Section IV-B2)

A Network Architecture

The Mask-RCNN architecture contains three stages as de-scribed below (see Figure 3)Backbone The first stage referred to as the backbone isused to extract features from the input image It consistsof a convolutional network combined with a feature-pyramidnetwork [42] thereby enabling multi-scale features to beextracted We use the first four blocks of ResNet-50 [43] asthe convolutional networkRegion Proposal Network (RPN) This is a convolutionalnetwork which scans the pyramid feature map generatedby the backbone network and generates rectangular regionscommonly called lsquoobject proposalsrsquo which are likely to containobjects of interest For each level of the feature pyramid andfor each spatial location at a given level a set of level-specificbounding boxes called anchors are generated The anchorstypically span a range of aspect ratios (eg 1 2 1 1 2 1)for flexibility in detection For each anchor the RPN networkpredicts (i) the probability of an object being present (lsquoobject-ness scorersquo) (ii) offset coordinates of a bounding box relative to

location of the anchor The generated bounding boxes are firstfiltered according to the lsquoobjectness scorersquo From boxes whichsurvive the filtering those that overlap with the underlyingobject above a certain threshold are chosen After applyingnon-maximal suppression to remove overlapping boxes withrelatively smaller objectness scores the final set of boxeswhich remain are termed lsquoobject proposalsrsquo or Regions-of-Interest (RoI)Multi-Task Branch Networks The RoIs obtained from RPNare warped into fixed dimensions and overlaid on feature mapsextracted from the backbone to obtain RoI-specific featuresThese features are fed to three parallel task sub-networks Thefirst sub-network maps these features to region labels (egHoleCharacter-Line-Segment) while the second sub-network maps the RoI features to bounding boxes The thirdsub-network is fully convolutional and maps the features tothe pixel mask of the underlying region Note that the abilityof the architecture to predict masks independently for eachRoI plays a crucial role in obtaining instance segmentationsAnother advantage is that it naturally addresses situationswhere annotations or predictions overlap

B Implementation Details

The dataset splits used for training validation and testphases can be seen in Table II All manuscript images areadaptively resized to ensure the width does not exceed 1024pixels The images are padded with zeros such that the inputto the deep network has spatial dimensions of 1024 times 1024The ground truth region masks are initially subjected to asimilar resizing procedure Subsequently they are downsizedto 28 times 28 in order to match output dimensions of the masksub-network

1) Training The network is initialized with weights ob-tained from a Mask R-CNN trained on the MS-COCO [44]dataset with a ResNet-50 backbone We found that this re-sults in faster convergence and stabler training compared tousing weights from a Mask-RCNN trained on ImageNet [45]or training from scratch Within the RPN network we usecustom-designed anchors of 5 different scales and with 3different aspect ratios Specifically we use the followingaspect ratios ndash 1113110 ndash keeping in mind the typicalspatial extents of the various region classes We also limit

Predicted region types

ROI Align

FC layer

FC layer

FC layer

Input Image

Feature Pyramid

Line

Region Proposals

Detected Bounding

boxesRPN

CLS

PB

P

Region Classifier

Bounding box regressor

Instance Mask Predictor

ResNet-50 Backbone

Page Boundary

Picture

Created by Unlicensed Version

Created by Unlicensed Version

Created by Unlicensed Version

Fig 3 The architecture adopted for Indic Manuscript Layout Parsing Refer to Section IV for details

Average IoU Average Per pixel AccuracyDataset darr H CLS PD PB CC P D LM BL

PIH minus 74179257 minus 86909637 52847485 60498221 523617 50295697 29454314BHOOMI 79299995 29074367 8721298 91099922 32504719 minus minus 38254998 minusCombined 79299995 57777479 8721298 88479744 45876537 60498221 523617 42935270 29454314

TABLE IV Class-wise average IoUs and per-pixel accuracies on the test set Refer to Table I for full names of abbreviatedregion types listed at top of the table

AP50 AP75 AP

PIH 7978 6011 4964Bhoomi 3688 1495 1800

Combined 6476 4430 3857

TABLE V AP at IoU thresholds 50 75 and overall APaveraged over IoU range for test set

the number of RoIs (lsquoobject proposalsrsquo) to 512 We usecategorical cross entropy loss LRPN for RPN classificationnetwork Within the task branches we use categorical crossentropy loss Lr for region classification branch smooth L1loss [46] (Lbb) for final bounding box prediction and per-pixel binary cross entropy loss Lmask for mask predictionThe total loss is a convex combination of these losses ieL = λRPNLRPN + λrLr + λbbLbb + λmaskLmask Theweighting factors (λs) are set to 1 However to ensure priorityfor our task of interest namely mask prediction we setλmask = 2 For optimization we use Stochastic GradientDescent (SGD) optimizer with a gradient norm clipping valueof 05 The batch size momentum and weight decay are setto 1 09 and 10minus3 respectively Given the relatively smallersize of our manuscript dataset compared to the photo dataset

(MS-COCO) used to originally train the base Mask R-CNNwe adopt a multi-stage training strategy For the first stage (30epochs) we train only the task branch sub-networks using alearning rate of 10minus3 while freezing weights in the rest of theoverall network This ensures that the task branches are fine-tuned for the types of regions contained in manuscript imagesFor the second stage (20 epochs) we additionally train stage-4 and up of the backbone ResNet-50 This enables extractionof appropriate semantic features from manuscript images Theomission of the initial 3 stages in the backbone for trainingis due to the fact that they provide generic re-usable low-level features To ensure priority coverage of hard-to-localizeregions we use focal loss [47] for mask generation For thefinal stage (15 epochs) we train the entire network using alearning rate of 10minus4

2) Inference During inference the images are rescaledand processed using the procedure described at the beginningof the subsection The number of RoIs retained after non-maximal suppression (NMS) from the RPN is set to 1000From these we choose the top 100 region detections withobjectness score exceeding 05 and feed the correspondingRoIs to the mask branch sub-network for mask generationIt is important to note that this strategy is different from theparallel generation of outputs and use of the task sub-networksduring training The generated masks are then binarized usingan empirically chosen threshold of 04 and rescaled to their

Fig 4 Ground truth annotations (left) and predicted instance segmentations (right) for test set images Note that we use coloredshading only to visualize individual region instances and not to color-code region types The region label abbreviations areshown alongside the regions CLS Character Line Segment PB Page Boundary H Hole BL Boundary Line CC Character Component PD Physical Degradation

original size using bilinear interpolation On these generatedmasks NMS with a threshold value of 05 is applied to obtainthe final set of predicted masks

C Evaluation

For quantitative evaluation we compute Average Precision(AP) for a particular IoU threshold a measure widely reportedin instance segmentation literature [44] [48] We specificallyreport AP50 and AP75 corresponding to AP at IoU thresholds50 and 75 respectively [41] In addition we report an overallscore by averaging AP at different IoU thresholds ranging from05 to 095 in steps of 005

The AP measure characterizes performance at documentlevel To characterize performance for each region type wereport two additional measures [25] ndash average class-wise IoU(cwIoU) and average class-wise per-pixel accuracy (cwAcc)Consider a fixed test document k Suppose there are ri regionsof class i and let IoUr denote the IoU score for one suchregion r ie 1 6 r 6 ri The per-class IoU score for class iand document k is computed as cwIoUd

i =sum

r IoUr

ri Suppose

there are Ni documents containing at least a single region ofclass i in ground-truth The overall per-class IoU score forclass i is computed as cwIoU i =

sumd cwIoUd

i

Ni In a similar

manner we define class-wise pixel accuracy pwAccdi at docu-ment level and average it across all the documents containingclass i ie cwAcci =

sumd pwAccdi

Ni Note that our approach

for computing class-wise scores prevents documents with arelatively larger number of class instances from dominating thescore and in this sense differs from existing approaches [25]

V RESULTS

We report quantitative results using the measures describedin Section IV-C Table IV reports Average Precision and TableV reports class-wise average IOUs and per-pixel accuraciesQualitative results can be viewed in Figure 4 Despite thechallenges posed by manuscripts our model performs rea-sonably well across a variety of classes As the qualitativeresults indicate the model predicts accurate masks for almostall the regions The results also indicate that our model handlesoverlap between Holes and Character line segments wellFrom ablative experiments we found that our choice of focalloss was crucial in obtaining accurate mask boundaries Unliketraditional semantic segmentation which would have produceda single blob-like region for line segments our instance-basedapproach isolates each text line separately Additionally theclear demarcation between Page-Boundary and background in-dicates that our system identifies semantically relevant regionsfor downstream analysis As the result at the bottom of Figure4 shows our system can even handle images with multiplepages thus removing the need for any pre-processing relatedto isolation of individual pages

From quantitative results we observe that Holes Characterline segments Page boundary and Pictures are parsed thebest while Physical degradations are difficult to parse dueto the relatively small footprint and inconsistent patterns indegradations The results show that performance for Penn

in Hand (PIH) documents is better compared to BHOOMImanuscripts We conjecture that the presence of closely spacedand unevenly written lines in latter is the cause In ourapproach two (or more) objects may share the same boundingbox in terms of overlap and it is not possible to determinewhich box to choose during mask prediction Consequentlyan underlying linersquos boundary may either end up not beingdetected or the predicted mask might be poorly localizedHowever this is not a systemic problem since our modelachieves good performance even for very dense BHOOMIdocument line layouts

VI CONCLUSION

Via this paper we propose Indiscapes the first dataset withlayout annotations for historical Indic manuscripts We believethat the availability of layout annotations will play a crucialrole in reducing the overall complexity for OCR and othertasks such as word-spotting style-and-content based retrievalIn the long-term we intend to expand the dataset not onlynumerically but also in terms of layout script and languagediversity As a significant contribution we have also adapted adeep-network based instance segmentation framework custommodified for fully automatic layout parsing Given the generalnature of our framework advances in instance segmentationapproaches can be leveraged thereby improving performanceover time Our proposed web-based annotator system althoughdesigned for Indic manuscripts is flexible and could be reusedfor similar manuscripts from Asian subcontinent We intend toexpand the capabilities of our annotator system in many usefulways For instance the layout estimated by our deep-networkcould be provided to annotators for correction thus reduc-ing annotation efforts Finally we plan to have our datasetinstance segmentation system and annotator system publiclyavailable This would enable large-scale data collection andautomated analysis efforts for Indic as well as other historicalAsian manuscripts The repositories related to the systemspresented in this paper and the Indiscapes dataset can beaccessed at httpsihdiaiiitacin

ACKNOWLEDGMENT

We would like to thank Dr Sai Susarla for enabling accessto the Bhoomi document collection We also thank PoreddyMourya Kumar Reddy Gollapudi Sai Vamsi Krishna for theircontributions related to dashboard and various annotators fortheir labelling efforts

REFERENCES

[1] C Reul M Dittrich and M Gruner ldquoCase study of a highly automatedlayout analysis and ocr of an incunabulumrsquoder heiligen lebenrsquo(1488)rdquoin Proc 2nd Intl Conf on Digital Access to Textual Cultural HeritageACM 2017 pp 155ndash160 1

[2] U Springmann and A Luedeling ldquoOcr of historical printings with anapplication to building diachronic corpora A case study using the ridgesherbal corpusrdquo Digital Humanities Quarterly no 2 2017 1

[3] F Simistira M Seuret N Eichenberger A Garz M Liwicki and R In-gold ldquoDiva-hisdb A precisely annotated large dataset of challengingmedieval manuscriptsrdquo in ICFHR IEEE 2016 pp 471ndash476 1

Character Line Segment Character Component Hole Page Boundary Library Marker Decorator Picture Physical Degradation Boundary Line(CLS) (CC) (H) (PB) (LM) (D) (P) (PD) (BL)

PIH 2401 494 minus 256 32 59 94 34 395BHOOMI 2440 210 565 316 133 minus minus 2078 minus

Combined 4841 704 565 572 165 59 94 2112 395

TABLE I Counts for various annotated region types in INDISCAPES dataset The abbreviations used for region types are givenbelow each region type

Train Validation Test Total

PIH 116 28 49 193BHOOMI 236 59 20 315

Total 352 87 69 508

TABLE II Dataset splits used for learning and inference

Script Source Document Count

Devanagari PIH 193Nandinagari BHOOMI 2

Telugu BHOOMI 75Grantha BHOOMI 238

TABLE III Scripts in the INDISCAPES dataset

BHOOMI an assorted collection of 315 images sourced frommultiple Oriental Research Institutes and libraries across IndiaAs with the first collection we chose a subset intended tomaximize the overall diversity of the dataset However thislatter set of images are characterized by a relatively inferiordocument quality presence of multiple languages and from alayout point of view predominantly contain long closely andirregularly spaced text lines binding holes and degradations(Figure 1) Though some document images contain multiplemanuscripts we do not attempt to split the image into mul-tiple pages While this poses a challenge for annotation andautomatic image parsing retaining such images in the dataseteliminates manualsemi-automatic intervention As our resultsshow our approach can successfully handle such multi-pagedocuments thereby making it truly an end-to-end system

Overall our dataset contains 508 annotated Indicmanuscripts Some salient aspects of the dataset can beviewed in Table I and a pictorial illustration of layout regionscan be viewed in Figure 4 Note that multiple regions canoverlap unlike existing historical document datasets whichtypically contain disjoint region annotations

For the rest of the section we discuss the challengesassociated with annotating Indic manuscripts (Section III-A)and our web-based annotation tool (Section III-B)

A Annotation Challenges

A variety of unique challenges exist in the context ofannotating Indic manuscript layouts The challenges arise fromthree major sources

Content The documents are written in a large variety ofIndic languages Some languages even exhibit intra-languagescript variations A large pool of annotators familiar with thelanguages and scripts present in the corpus is required toensure proper annotation of lines and character componentsLayout Unlike some of the existing datasets Indicmanuscripts contain non-textual elements such as color pic-tures tables and document decorations These elements arefrequently interspersed with text in non-standard layouts Inmany cases the manuscripts contain one or more physicalholes designed for a thread-like material to pass throughand bind the leaves together as a book Such holes vary interms of spatial location count and hole diameter When theholes are present in the middle of the document they causea break in the contiguity of lines In some documents theline contiguity is broken by a lsquovirtualrsquo hole-like gap possiblyintended for creation of the punched hole at a future timeIn many cases the separation between lines is extremelysmall The handwritten nature of these documents and thesurface material result in extremely uneven lines necessitatingmeticulous and slow annotation If multiple manuscript pagesare present the stacking order could be horizontal or verticalOverall the sheer variety in layout elements poses a significantchallenge not only for annotation but also for automatedlayout parsingDegradations Historical Indic manuscripts tend to be inher-ently fragile and prone to damage due to various sourcesndash wood-and-leaf-boring insects humidity seepage improperstorage and handling etc While some degradations cause theedges of the document to become frayed others manifest asirregularly shaped perforations in the document interior It maybe important to identify such degradations before attemptinglexically-focused tasks such as OCR or word-spotting

B Annotation Tool

Keeping the aforementioned challenges in mind we in-troduce a new browser-based annotation tool (see Figure2) The tool is designed to operate both stand-alone andas a web-service The web-service mode enables featuressuch as distributed parallel sessions by registered annotatorsdashboard-based live session monitoring and a wide varietyof annotation-related analytics On the front-end a freehandregion option is provided alongside the usual rectangle andpolygon to enable maximum annotation flexibility The web-service version also features a lsquoCorrection-modersquo which en-ables annotators to correct existing annotations from previousannotators Additionally the tool has been designed to enable

Fig 2 Screenshots of our web-based annotator (left) and analytics dashboard (right)

lexical (text) annotations in future

IV INDIC MANUSCRIPT LAYOUT PARSING

To succeed at layout parsing of manuscripts we requirea system which can accurately localize various types ofregions (eg text lines isolated character components physicaldegradation pictures holes) More importantly we require asystem which can isolate individual instances of each region(eg multiple text lines) in the manuscript image Also in ourcase the annotation regions for manuscripts are not disjointand can overlap (eg The annotation region for a text line canoverlap with the annotation region of a hole (see Figure 4))Therefore we require a system which can accommodate suchoverlaps To meet all of these requirements we model ourproblem as one of semantic instance-level segmentation andemploy the Mask R-CNN [41] architecture which has provento be very effective at the task of object-instance segmenta-tion in photos Next we briefly describe the Mask R-CNNarchitecture and our modifications of the same Subsequentlywe provide details related to implementation (Section IV-B)model training (Section IV-B1) and inference (Section IV-B2)

A Network Architecture

The Mask-RCNN architecture contains three stages as de-scribed below (see Figure 3)Backbone The first stage referred to as the backbone isused to extract features from the input image It consistsof a convolutional network combined with a feature-pyramidnetwork [42] thereby enabling multi-scale features to beextracted We use the first four blocks of ResNet-50 [43] asthe convolutional networkRegion Proposal Network (RPN) This is a convolutionalnetwork which scans the pyramid feature map generatedby the backbone network and generates rectangular regionscommonly called lsquoobject proposalsrsquo which are likely to containobjects of interest For each level of the feature pyramid andfor each spatial location at a given level a set of level-specificbounding boxes called anchors are generated The anchorstypically span a range of aspect ratios (eg 1 2 1 1 2 1)for flexibility in detection For each anchor the RPN networkpredicts (i) the probability of an object being present (lsquoobject-ness scorersquo) (ii) offset coordinates of a bounding box relative to

location of the anchor The generated bounding boxes are firstfiltered according to the lsquoobjectness scorersquo From boxes whichsurvive the filtering those that overlap with the underlyingobject above a certain threshold are chosen After applyingnon-maximal suppression to remove overlapping boxes withrelatively smaller objectness scores the final set of boxeswhich remain are termed lsquoobject proposalsrsquo or Regions-of-Interest (RoI)Multi-Task Branch Networks The RoIs obtained from RPNare warped into fixed dimensions and overlaid on feature mapsextracted from the backbone to obtain RoI-specific featuresThese features are fed to three parallel task sub-networks Thefirst sub-network maps these features to region labels (egHoleCharacter-Line-Segment) while the second sub-network maps the RoI features to bounding boxes The thirdsub-network is fully convolutional and maps the features tothe pixel mask of the underlying region Note that the abilityof the architecture to predict masks independently for eachRoI plays a crucial role in obtaining instance segmentationsAnother advantage is that it naturally addresses situationswhere annotations or predictions overlap

B Implementation Details

The dataset splits used for training validation and testphases can be seen in Table II All manuscript images areadaptively resized to ensure the width does not exceed 1024pixels The images are padded with zeros such that the inputto the deep network has spatial dimensions of 1024 times 1024The ground truth region masks are initially subjected to asimilar resizing procedure Subsequently they are downsizedto 28 times 28 in order to match output dimensions of the masksub-network

1) Training The network is initialized with weights ob-tained from a Mask R-CNN trained on the MS-COCO [44]dataset with a ResNet-50 backbone We found that this re-sults in faster convergence and stabler training compared tousing weights from a Mask-RCNN trained on ImageNet [45]or training from scratch Within the RPN network we usecustom-designed anchors of 5 different scales and with 3different aspect ratios Specifically we use the followingaspect ratios ndash 1113110 ndash keeping in mind the typicalspatial extents of the various region classes We also limit

Predicted region types

ROI Align

FC layer

FC layer

FC layer

Input Image

Feature Pyramid

Line

Region Proposals

Detected Bounding

boxesRPN

CLS

PB

P

Region Classifier

Bounding box regressor

Instance Mask Predictor

ResNet-50 Backbone

Page Boundary

Picture

Created by Unlicensed Version

Created by Unlicensed Version

Created by Unlicensed Version

Fig 3 The architecture adopted for Indic Manuscript Layout Parsing Refer to Section IV for details

Average IoU Average Per pixel AccuracyDataset darr H CLS PD PB CC P D LM BL

PIH minus 74179257 minus 86909637 52847485 60498221 523617 50295697 29454314BHOOMI 79299995 29074367 8721298 91099922 32504719 minus minus 38254998 minusCombined 79299995 57777479 8721298 88479744 45876537 60498221 523617 42935270 29454314

TABLE IV Class-wise average IoUs and per-pixel accuracies on the test set Refer to Table I for full names of abbreviatedregion types listed at top of the table

AP50 AP75 AP

PIH 7978 6011 4964Bhoomi 3688 1495 1800

Combined 6476 4430 3857

TABLE V AP at IoU thresholds 50 75 and overall APaveraged over IoU range for test set

the number of RoIs (lsquoobject proposalsrsquo) to 512 We usecategorical cross entropy loss LRPN for RPN classificationnetwork Within the task branches we use categorical crossentropy loss Lr for region classification branch smooth L1loss [46] (Lbb) for final bounding box prediction and per-pixel binary cross entropy loss Lmask for mask predictionThe total loss is a convex combination of these losses ieL = λRPNLRPN + λrLr + λbbLbb + λmaskLmask Theweighting factors (λs) are set to 1 However to ensure priorityfor our task of interest namely mask prediction we setλmask = 2 For optimization we use Stochastic GradientDescent (SGD) optimizer with a gradient norm clipping valueof 05 The batch size momentum and weight decay are setto 1 09 and 10minus3 respectively Given the relatively smallersize of our manuscript dataset compared to the photo dataset

(MS-COCO) used to originally train the base Mask R-CNNwe adopt a multi-stage training strategy For the first stage (30epochs) we train only the task branch sub-networks using alearning rate of 10minus3 while freezing weights in the rest of theoverall network This ensures that the task branches are fine-tuned for the types of regions contained in manuscript imagesFor the second stage (20 epochs) we additionally train stage-4 and up of the backbone ResNet-50 This enables extractionof appropriate semantic features from manuscript images Theomission of the initial 3 stages in the backbone for trainingis due to the fact that they provide generic re-usable low-level features To ensure priority coverage of hard-to-localizeregions we use focal loss [47] for mask generation For thefinal stage (15 epochs) we train the entire network using alearning rate of 10minus4

2) Inference During inference the images are rescaledand processed using the procedure described at the beginningof the subsection The number of RoIs retained after non-maximal suppression (NMS) from the RPN is set to 1000From these we choose the top 100 region detections withobjectness score exceeding 05 and feed the correspondingRoIs to the mask branch sub-network for mask generationIt is important to note that this strategy is different from theparallel generation of outputs and use of the task sub-networksduring training The generated masks are then binarized usingan empirically chosen threshold of 04 and rescaled to their

Fig 4 Ground truth annotations (left) and predicted instance segmentations (right) for test set images Note that we use coloredshading only to visualize individual region instances and not to color-code region types The region label abbreviations areshown alongside the regions CLS Character Line Segment PB Page Boundary H Hole BL Boundary Line CC Character Component PD Physical Degradation

original size using bilinear interpolation On these generatedmasks NMS with a threshold value of 05 is applied to obtainthe final set of predicted masks

C Evaluation

For quantitative evaluation we compute Average Precision(AP) for a particular IoU threshold a measure widely reportedin instance segmentation literature [44] [48] We specificallyreport AP50 and AP75 corresponding to AP at IoU thresholds50 and 75 respectively [41] In addition we report an overallscore by averaging AP at different IoU thresholds ranging from05 to 095 in steps of 005

The AP measure characterizes performance at documentlevel To characterize performance for each region type wereport two additional measures [25] ndash average class-wise IoU(cwIoU) and average class-wise per-pixel accuracy (cwAcc)Consider a fixed test document k Suppose there are ri regionsof class i and let IoUr denote the IoU score for one suchregion r ie 1 6 r 6 ri The per-class IoU score for class iand document k is computed as cwIoUd

i =sum

r IoUr

ri Suppose

there are Ni documents containing at least a single region ofclass i in ground-truth The overall per-class IoU score forclass i is computed as cwIoU i =

sumd cwIoUd

i

Ni In a similar

manner we define class-wise pixel accuracy pwAccdi at docu-ment level and average it across all the documents containingclass i ie cwAcci =

sumd pwAccdi

Ni Note that our approach

for computing class-wise scores prevents documents with arelatively larger number of class instances from dominating thescore and in this sense differs from existing approaches [25]

V RESULTS

We report quantitative results using the measures describedin Section IV-C Table IV reports Average Precision and TableV reports class-wise average IOUs and per-pixel accuraciesQualitative results can be viewed in Figure 4 Despite thechallenges posed by manuscripts our model performs rea-sonably well across a variety of classes As the qualitativeresults indicate the model predicts accurate masks for almostall the regions The results also indicate that our model handlesoverlap between Holes and Character line segments wellFrom ablative experiments we found that our choice of focalloss was crucial in obtaining accurate mask boundaries Unliketraditional semantic segmentation which would have produceda single blob-like region for line segments our instance-basedapproach isolates each text line separately Additionally theclear demarcation between Page-Boundary and background in-dicates that our system identifies semantically relevant regionsfor downstream analysis As the result at the bottom of Figure4 shows our system can even handle images with multiplepages thus removing the need for any pre-processing relatedto isolation of individual pages

From quantitative results we observe that Holes Characterline segments Page boundary and Pictures are parsed thebest while Physical degradations are difficult to parse dueto the relatively small footprint and inconsistent patterns indegradations The results show that performance for Penn

in Hand (PIH) documents is better compared to BHOOMImanuscripts We conjecture that the presence of closely spacedand unevenly written lines in latter is the cause In ourapproach two (or more) objects may share the same boundingbox in terms of overlap and it is not possible to determinewhich box to choose during mask prediction Consequentlyan underlying linersquos boundary may either end up not beingdetected or the predicted mask might be poorly localizedHowever this is not a systemic problem since our modelachieves good performance even for very dense BHOOMIdocument line layouts

VI CONCLUSION

Via this paper we propose Indiscapes the first dataset withlayout annotations for historical Indic manuscripts We believethat the availability of layout annotations will play a crucialrole in reducing the overall complexity for OCR and othertasks such as word-spotting style-and-content based retrievalIn the long-term we intend to expand the dataset not onlynumerically but also in terms of layout script and languagediversity As a significant contribution we have also adapted adeep-network based instance segmentation framework custommodified for fully automatic layout parsing Given the generalnature of our framework advances in instance segmentationapproaches can be leveraged thereby improving performanceover time Our proposed web-based annotator system althoughdesigned for Indic manuscripts is flexible and could be reusedfor similar manuscripts from Asian subcontinent We intend toexpand the capabilities of our annotator system in many usefulways For instance the layout estimated by our deep-networkcould be provided to annotators for correction thus reduc-ing annotation efforts Finally we plan to have our datasetinstance segmentation system and annotator system publiclyavailable This would enable large-scale data collection andautomated analysis efforts for Indic as well as other historicalAsian manuscripts The repositories related to the systemspresented in this paper and the Indiscapes dataset can beaccessed at httpsihdiaiiitacin

ACKNOWLEDGMENT

We would like to thank Dr Sai Susarla for enabling accessto the Bhoomi document collection We also thank PoreddyMourya Kumar Reddy Gollapudi Sai Vamsi Krishna for theircontributions related to dashboard and various annotators fortheir labelling efforts

REFERENCES

[1] C Reul M Dittrich and M Gruner ldquoCase study of a highly automatedlayout analysis and ocr of an incunabulumrsquoder heiligen lebenrsquo(1488)rdquoin Proc 2nd Intl Conf on Digital Access to Textual Cultural HeritageACM 2017 pp 155ndash160 1

[2] U Springmann and A Luedeling ldquoOcr of historical printings with anapplication to building diachronic corpora A case study using the ridgesherbal corpusrdquo Digital Humanities Quarterly no 2 2017 1

[3] F Simistira M Seuret N Eichenberger A Garz M Liwicki and R In-gold ldquoDiva-hisdb A precisely annotated large dataset of challengingmedieval manuscriptsrdquo in ICFHR IEEE 2016 pp 471ndash476 1

Fig 2 Screenshots of our web-based annotator (left) and analytics dashboard (right)

lexical (text) annotations in future

IV INDIC MANUSCRIPT LAYOUT PARSING

To succeed at layout parsing of manuscripts we requirea system which can accurately localize various types ofregions (eg text lines isolated character components physicaldegradation pictures holes) More importantly we require asystem which can isolate individual instances of each region(eg multiple text lines) in the manuscript image Also in ourcase the annotation regions for manuscripts are not disjointand can overlap (eg The annotation region for a text line canoverlap with the annotation region of a hole (see Figure 4))Therefore we require a system which can accommodate suchoverlaps To meet all of these requirements we model ourproblem as one of semantic instance-level segmentation andemploy the Mask R-CNN [41] architecture which has provento be very effective at the task of object-instance segmenta-tion in photos Next we briefly describe the Mask R-CNNarchitecture and our modifications of the same Subsequentlywe provide details related to implementation (Section IV-B)model training (Section IV-B1) and inference (Section IV-B2)

A Network Architecture

The Mask-RCNN architecture contains three stages as de-scribed below (see Figure 3)Backbone The first stage referred to as the backbone isused to extract features from the input image It consistsof a convolutional network combined with a feature-pyramidnetwork [42] thereby enabling multi-scale features to beextracted We use the first four blocks of ResNet-50 [43] asthe convolutional networkRegion Proposal Network (RPN) This is a convolutionalnetwork which scans the pyramid feature map generatedby the backbone network and generates rectangular regionscommonly called lsquoobject proposalsrsquo which are likely to containobjects of interest For each level of the feature pyramid andfor each spatial location at a given level a set of level-specificbounding boxes called anchors are generated The anchorstypically span a range of aspect ratios (eg 1 2 1 1 2 1)for flexibility in detection For each anchor the RPN networkpredicts (i) the probability of an object being present (lsquoobject-ness scorersquo) (ii) offset coordinates of a bounding box relative to

location of the anchor The generated bounding boxes are firstfiltered according to the lsquoobjectness scorersquo From boxes whichsurvive the filtering those that overlap with the underlyingobject above a certain threshold are chosen After applyingnon-maximal suppression to remove overlapping boxes withrelatively smaller objectness scores the final set of boxeswhich remain are termed lsquoobject proposalsrsquo or Regions-of-Interest (RoI)Multi-Task Branch Networks The RoIs obtained from RPNare warped into fixed dimensions and overlaid on feature mapsextracted from the backbone to obtain RoI-specific featuresThese features are fed to three parallel task sub-networks Thefirst sub-network maps these features to region labels (egHoleCharacter-Line-Segment) while the second sub-network maps the RoI features to bounding boxes The thirdsub-network is fully convolutional and maps the features tothe pixel mask of the underlying region Note that the abilityof the architecture to predict masks independently for eachRoI plays a crucial role in obtaining instance segmentationsAnother advantage is that it naturally addresses situationswhere annotations or predictions overlap

B Implementation Details

The dataset splits used for training validation and testphases can be seen in Table II All manuscript images areadaptively resized to ensure the width does not exceed 1024pixels The images are padded with zeros such that the inputto the deep network has spatial dimensions of 1024 times 1024The ground truth region masks are initially subjected to asimilar resizing procedure Subsequently they are downsizedto 28 times 28 in order to match output dimensions of the masksub-network

1) Training The network is initialized with weights ob-tained from a Mask R-CNN trained on the MS-COCO [44]dataset with a ResNet-50 backbone We found that this re-sults in faster convergence and stabler training compared tousing weights from a Mask-RCNN trained on ImageNet [45]or training from scratch Within the RPN network we usecustom-designed anchors of 5 different scales and with 3different aspect ratios Specifically we use the followingaspect ratios ndash 1113110 ndash keeping in mind the typicalspatial extents of the various region classes We also limit

Predicted region types

ROI Align

FC layer

FC layer

FC layer

Input Image

Feature Pyramid

Line

Region Proposals

Detected Bounding

boxesRPN

CLS

PB

P

Region Classifier

Bounding box regressor

Instance Mask Predictor

ResNet-50 Backbone

Page Boundary

Picture

Created by Unlicensed Version

Created by Unlicensed Version

Created by Unlicensed Version

Fig 3 The architecture adopted for Indic Manuscript Layout Parsing Refer to Section IV for details

Average IoU Average Per pixel AccuracyDataset darr H CLS PD PB CC P D LM BL

PIH minus 74179257 minus 86909637 52847485 60498221 523617 50295697 29454314BHOOMI 79299995 29074367 8721298 91099922 32504719 minus minus 38254998 minusCombined 79299995 57777479 8721298 88479744 45876537 60498221 523617 42935270 29454314

TABLE IV Class-wise average IoUs and per-pixel accuracies on the test set Refer to Table I for full names of abbreviatedregion types listed at top of the table

AP50 AP75 AP

PIH 7978 6011 4964Bhoomi 3688 1495 1800

Combined 6476 4430 3857

TABLE V AP at IoU thresholds 50 75 and overall APaveraged over IoU range for test set

the number of RoIs (lsquoobject proposalsrsquo) to 512 We usecategorical cross entropy loss LRPN for RPN classificationnetwork Within the task branches we use categorical crossentropy loss Lr for region classification branch smooth L1loss [46] (Lbb) for final bounding box prediction and per-pixel binary cross entropy loss Lmask for mask predictionThe total loss is a convex combination of these losses ieL = λRPNLRPN + λrLr + λbbLbb + λmaskLmask Theweighting factors (λs) are set to 1 However to ensure priorityfor our task of interest namely mask prediction we setλmask = 2 For optimization we use Stochastic GradientDescent (SGD) optimizer with a gradient norm clipping valueof 05 The batch size momentum and weight decay are setto 1 09 and 10minus3 respectively Given the relatively smallersize of our manuscript dataset compared to the photo dataset

(MS-COCO) used to originally train the base Mask R-CNNwe adopt a multi-stage training strategy For the first stage (30epochs) we train only the task branch sub-networks using alearning rate of 10minus3 while freezing weights in the rest of theoverall network This ensures that the task branches are fine-tuned for the types of regions contained in manuscript imagesFor the second stage (20 epochs) we additionally train stage-4 and up of the backbone ResNet-50 This enables extractionof appropriate semantic features from manuscript images Theomission of the initial 3 stages in the backbone for trainingis due to the fact that they provide generic re-usable low-level features To ensure priority coverage of hard-to-localizeregions we use focal loss [47] for mask generation For thefinal stage (15 epochs) we train the entire network using alearning rate of 10minus4

2) Inference During inference the images are rescaledand processed using the procedure described at the beginningof the subsection The number of RoIs retained after non-maximal suppression (NMS) from the RPN is set to 1000From these we choose the top 100 region detections withobjectness score exceeding 05 and feed the correspondingRoIs to the mask branch sub-network for mask generationIt is important to note that this strategy is different from theparallel generation of outputs and use of the task sub-networksduring training The generated masks are then binarized usingan empirically chosen threshold of 04 and rescaled to their

Fig 4 Ground truth annotations (left) and predicted instance segmentations (right) for test set images Note that we use coloredshading only to visualize individual region instances and not to color-code region types The region label abbreviations areshown alongside the regions CLS Character Line Segment PB Page Boundary H Hole BL Boundary Line CC Character Component PD Physical Degradation

original size using bilinear interpolation On these generatedmasks NMS with a threshold value of 05 is applied to obtainthe final set of predicted masks

C Evaluation

For quantitative evaluation we compute Average Precision(AP) for a particular IoU threshold a measure widely reportedin instance segmentation literature [44] [48] We specificallyreport AP50 and AP75 corresponding to AP at IoU thresholds50 and 75 respectively [41] In addition we report an overallscore by averaging AP at different IoU thresholds ranging from05 to 095 in steps of 005

The AP measure characterizes performance at documentlevel To characterize performance for each region type wereport two additional measures [25] ndash average class-wise IoU(cwIoU) and average class-wise per-pixel accuracy (cwAcc)Consider a fixed test document k Suppose there are ri regionsof class i and let IoUr denote the IoU score for one suchregion r ie 1 6 r 6 ri The per-class IoU score for class iand document k is computed as cwIoUd

i =sum

r IoUr

ri Suppose

there are Ni documents containing at least a single region ofclass i in ground-truth The overall per-class IoU score forclass i is computed as cwIoU i =

sumd cwIoUd

i

Ni In a similar

manner we define class-wise pixel accuracy pwAccdi at docu-ment level and average it across all the documents containingclass i ie cwAcci =

sumd pwAccdi

Ni Note that our approach

for computing class-wise scores prevents documents with arelatively larger number of class instances from dominating thescore and in this sense differs from existing approaches [25]

V RESULTS

We report quantitative results using the measures describedin Section IV-C Table IV reports Average Precision and TableV reports class-wise average IOUs and per-pixel accuraciesQualitative results can be viewed in Figure 4 Despite thechallenges posed by manuscripts our model performs rea-sonably well across a variety of classes As the qualitativeresults indicate the model predicts accurate masks for almostall the regions The results also indicate that our model handlesoverlap between Holes and Character line segments wellFrom ablative experiments we found that our choice of focalloss was crucial in obtaining accurate mask boundaries Unliketraditional semantic segmentation which would have produceda single blob-like region for line segments our instance-basedapproach isolates each text line separately Additionally theclear demarcation between Page-Boundary and background in-dicates that our system identifies semantically relevant regionsfor downstream analysis As the result at the bottom of Figure4 shows our system can even handle images with multiplepages thus removing the need for any pre-processing relatedto isolation of individual pages

From quantitative results we observe that Holes Characterline segments Page boundary and Pictures are parsed thebest while Physical degradations are difficult to parse dueto the relatively small footprint and inconsistent patterns indegradations The results show that performance for Penn

in Hand (PIH) documents is better compared to BHOOMImanuscripts We conjecture that the presence of closely spacedand unevenly written lines in latter is the cause In ourapproach two (or more) objects may share the same boundingbox in terms of overlap and it is not possible to determinewhich box to choose during mask prediction Consequentlyan underlying linersquos boundary may either end up not beingdetected or the predicted mask might be poorly localizedHowever this is not a systemic problem since our modelachieves good performance even for very dense BHOOMIdocument line layouts

VI CONCLUSION

Via this paper we propose Indiscapes the first dataset withlayout annotations for historical Indic manuscripts We believethat the availability of layout annotations will play a crucialrole in reducing the overall complexity for OCR and othertasks such as word-spotting style-and-content based retrievalIn the long-term we intend to expand the dataset not onlynumerically but also in terms of layout script and languagediversity As a significant contribution we have also adapted adeep-network based instance segmentation framework custommodified for fully automatic layout parsing Given the generalnature of our framework advances in instance segmentationapproaches can be leveraged thereby improving performanceover time Our proposed web-based annotator system althoughdesigned for Indic manuscripts is flexible and could be reusedfor similar manuscripts from Asian subcontinent We intend toexpand the capabilities of our annotator system in many usefulways For instance the layout estimated by our deep-networkcould be provided to annotators for correction thus reduc-ing annotation efforts Finally we plan to have our datasetinstance segmentation system and annotator system publiclyavailable This would enable large-scale data collection andautomated analysis efforts for Indic as well as other historicalAsian manuscripts The repositories related to the systemspresented in this paper and the Indiscapes dataset can beaccessed at httpsihdiaiiitacin

ACKNOWLEDGMENT

We would like to thank Dr Sai Susarla for enabling accessto the Bhoomi document collection We also thank PoreddyMourya Kumar Reddy Gollapudi Sai Vamsi Krishna for theircontributions related to dashboard and various annotators fortheir labelling efforts

REFERENCES

[1] C Reul M Dittrich and M Gruner ldquoCase study of a highly automatedlayout analysis and ocr of an incunabulumrsquoder heiligen lebenrsquo(1488)rdquoin Proc 2nd Intl Conf on Digital Access to Textual Cultural HeritageACM 2017 pp 155ndash160 1

[2] U Springmann and A Luedeling ldquoOcr of historical printings with anapplication to building diachronic corpora A case study using the ridgesherbal corpusrdquo Digital Humanities Quarterly no 2 2017 1

[3] F Simistira M Seuret N Eichenberger A Garz M Liwicki and R In-gold ldquoDiva-hisdb A precisely annotated large dataset of challengingmedieval manuscriptsrdquo in ICFHR IEEE 2016 pp 471ndash476 1

Predicted region types

ROI Align

FC layer

FC layer

FC layer

Input Image

Feature Pyramid

Line

Region Proposals

Detected Bounding

boxesRPN

CLS

PB

P

Region Classifier

Bounding box regressor

Instance Mask Predictor

ResNet-50 Backbone

Page Boundary

Picture

Created by Unlicensed Version

Created by Unlicensed Version

Created by Unlicensed Version

Fig 3 The architecture adopted for Indic Manuscript Layout Parsing Refer to Section IV for details

Average IoU Average Per pixel AccuracyDataset darr H CLS PD PB CC P D LM BL

PIH minus 74179257 minus 86909637 52847485 60498221 523617 50295697 29454314BHOOMI 79299995 29074367 8721298 91099922 32504719 minus minus 38254998 minusCombined 79299995 57777479 8721298 88479744 45876537 60498221 523617 42935270 29454314

TABLE IV Class-wise average IoUs and per-pixel accuracies on the test set Refer to Table I for full names of abbreviatedregion types listed at top of the table

AP50 AP75 AP

PIH 7978 6011 4964Bhoomi 3688 1495 1800

Combined 6476 4430 3857

TABLE V AP at IoU thresholds 50 75 and overall APaveraged over IoU range for test set

the number of RoIs (lsquoobject proposalsrsquo) to 512 We usecategorical cross entropy loss LRPN for RPN classificationnetwork Within the task branches we use categorical crossentropy loss Lr for region classification branch smooth L1loss [46] (Lbb) for final bounding box prediction and per-pixel binary cross entropy loss Lmask for mask predictionThe total loss is a convex combination of these losses ieL = λRPNLRPN + λrLr + λbbLbb + λmaskLmask Theweighting factors (λs) are set to 1 However to ensure priorityfor our task of interest namely mask prediction we setλmask = 2 For optimization we use Stochastic GradientDescent (SGD) optimizer with a gradient norm clipping valueof 05 The batch size momentum and weight decay are setto 1 09 and 10minus3 respectively Given the relatively smallersize of our manuscript dataset compared to the photo dataset

(MS-COCO) used to originally train the base Mask R-CNNwe adopt a multi-stage training strategy For the first stage (30epochs) we train only the task branch sub-networks using alearning rate of 10minus3 while freezing weights in the rest of theoverall network This ensures that the task branches are fine-tuned for the types of regions contained in manuscript imagesFor the second stage (20 epochs) we additionally train stage-4 and up of the backbone ResNet-50 This enables extractionof appropriate semantic features from manuscript images Theomission of the initial 3 stages in the backbone for trainingis due to the fact that they provide generic re-usable low-level features To ensure priority coverage of hard-to-localizeregions we use focal loss [47] for mask generation For thefinal stage (15 epochs) we train the entire network using alearning rate of 10minus4

2) Inference During inference the images are rescaledand processed using the procedure described at the beginningof the subsection The number of RoIs retained after non-maximal suppression (NMS) from the RPN is set to 1000From these we choose the top 100 region detections withobjectness score exceeding 05 and feed the correspondingRoIs to the mask branch sub-network for mask generationIt is important to note that this strategy is different from theparallel generation of outputs and use of the task sub-networksduring training The generated masks are then binarized usingan empirically chosen threshold of 04 and rescaled to their

Fig 4 Ground truth annotations (left) and predicted instance segmentations (right) for test set images Note that we use coloredshading only to visualize individual region instances and not to color-code region types The region label abbreviations areshown alongside the regions CLS Character Line Segment PB Page Boundary H Hole BL Boundary Line CC Character Component PD Physical Degradation

original size using bilinear interpolation On these generatedmasks NMS with a threshold value of 05 is applied to obtainthe final set of predicted masks

C Evaluation

For quantitative evaluation we compute Average Precision(AP) for a particular IoU threshold a measure widely reportedin instance segmentation literature [44] [48] We specificallyreport AP50 and AP75 corresponding to AP at IoU thresholds50 and 75 respectively [41] In addition we report an overallscore by averaging AP at different IoU thresholds ranging from05 to 095 in steps of 005

The AP measure characterizes performance at documentlevel To characterize performance for each region type wereport two additional measures [25] ndash average class-wise IoU(cwIoU) and average class-wise per-pixel accuracy (cwAcc)Consider a fixed test document k Suppose there are ri regionsof class i and let IoUr denote the IoU score for one suchregion r ie 1 6 r 6 ri The per-class IoU score for class iand document k is computed as cwIoUd

i =sum

r IoUr

ri Suppose

there are Ni documents containing at least a single region ofclass i in ground-truth The overall per-class IoU score forclass i is computed as cwIoU i =

sumd cwIoUd

i

Ni In a similar

manner we define class-wise pixel accuracy pwAccdi at docu-ment level and average it across all the documents containingclass i ie cwAcci =

sumd pwAccdi

Ni Note that our approach

for computing class-wise scores prevents documents with arelatively larger number of class instances from dominating thescore and in this sense differs from existing approaches [25]

V RESULTS

We report quantitative results using the measures describedin Section IV-C Table IV reports Average Precision and TableV reports class-wise average IOUs and per-pixel accuraciesQualitative results can be viewed in Figure 4 Despite thechallenges posed by manuscripts our model performs rea-sonably well across a variety of classes As the qualitativeresults indicate the model predicts accurate masks for almostall the regions The results also indicate that our model handlesoverlap between Holes and Character line segments wellFrom ablative experiments we found that our choice of focalloss was crucial in obtaining accurate mask boundaries Unliketraditional semantic segmentation which would have produceda single blob-like region for line segments our instance-basedapproach isolates each text line separately Additionally theclear demarcation between Page-Boundary and background in-dicates that our system identifies semantically relevant regionsfor downstream analysis As the result at the bottom of Figure4 shows our system can even handle images with multiplepages thus removing the need for any pre-processing relatedto isolation of individual pages

From quantitative results we observe that Holes Characterline segments Page boundary and Pictures are parsed thebest while Physical degradations are difficult to parse dueto the relatively small footprint and inconsistent patterns indegradations The results show that performance for Penn

in Hand (PIH) documents is better compared to BHOOMImanuscripts We conjecture that the presence of closely spacedand unevenly written lines in latter is the cause In ourapproach two (or more) objects may share the same boundingbox in terms of overlap and it is not possible to determinewhich box to choose during mask prediction Consequentlyan underlying linersquos boundary may either end up not beingdetected or the predicted mask might be poorly localizedHowever this is not a systemic problem since our modelachieves good performance even for very dense BHOOMIdocument line layouts

VI CONCLUSION

Via this paper we propose Indiscapes the first dataset withlayout annotations for historical Indic manuscripts We believethat the availability of layout annotations will play a crucialrole in reducing the overall complexity for OCR and othertasks such as word-spotting style-and-content based retrievalIn the long-term we intend to expand the dataset not onlynumerically but also in terms of layout script and languagediversity As a significant contribution we have also adapted adeep-network based instance segmentation framework custommodified for fully automatic layout parsing Given the generalnature of our framework advances in instance segmentationapproaches can be leveraged thereby improving performanceover time Our proposed web-based annotator system althoughdesigned for Indic manuscripts is flexible and could be reusedfor similar manuscripts from Asian subcontinent We intend toexpand the capabilities of our annotator system in many usefulways For instance the layout estimated by our deep-networkcould be provided to annotators for correction thus reduc-ing annotation efforts Finally we plan to have our datasetinstance segmentation system and annotator system publiclyavailable This would enable large-scale data collection andautomated analysis efforts for Indic as well as other historicalAsian manuscripts The repositories related to the systemspresented in this paper and the Indiscapes dataset can beaccessed at httpsihdiaiiitacin

ACKNOWLEDGMENT

We would like to thank Dr Sai Susarla for enabling accessto the Bhoomi document collection We also thank PoreddyMourya Kumar Reddy Gollapudi Sai Vamsi Krishna for theircontributions related to dashboard and various annotators fortheir labelling efforts

REFERENCES

[1] C Reul M Dittrich and M Gruner ldquoCase study of a highly automatedlayout analysis and ocr of an incunabulumrsquoder heiligen lebenrsquo(1488)rdquoin Proc 2nd Intl Conf on Digital Access to Textual Cultural HeritageACM 2017 pp 155ndash160 1

[2] U Springmann and A Luedeling ldquoOcr of historical printings with anapplication to building diachronic corpora A case study using the ridgesherbal corpusrdquo Digital Humanities Quarterly no 2 2017 1

[3] F Simistira M Seuret N Eichenberger A Garz M Liwicki and R In-gold ldquoDiva-hisdb A precisely annotated large dataset of challengingmedieval manuscriptsrdquo in ICFHR IEEE 2016 pp 471ndash476 1

Fig 4 Ground truth annotations (left) and predicted instance segmentations (right) for test set images Note that we use coloredshading only to visualize individual region instances and not to color-code region types The region label abbreviations areshown alongside the regions CLS Character Line Segment PB Page Boundary H Hole BL Boundary Line CC Character Component PD Physical Degradation

original size using bilinear interpolation On these generatedmasks NMS with a threshold value of 05 is applied to obtainthe final set of predicted masks

C Evaluation

For quantitative evaluation we compute Average Precision(AP) for a particular IoU threshold a measure widely reportedin instance segmentation literature [44] [48] We specificallyreport AP50 and AP75 corresponding to AP at IoU thresholds50 and 75 respectively [41] In addition we report an overallscore by averaging AP at different IoU thresholds ranging from05 to 095 in steps of 005

The AP measure characterizes performance at documentlevel To characterize performance for each region type wereport two additional measures [25] ndash average class-wise IoU(cwIoU) and average class-wise per-pixel accuracy (cwAcc)Consider a fixed test document k Suppose there are ri regionsof class i and let IoUr denote the IoU score for one suchregion r ie 1 6 r 6 ri The per-class IoU score for class iand document k is computed as cwIoUd

i =sum

r IoUr

ri Suppose

there are Ni documents containing at least a single region ofclass i in ground-truth The overall per-class IoU score forclass i is computed as cwIoU i =

sumd cwIoUd

i

Ni In a similar

manner we define class-wise pixel accuracy pwAccdi at docu-ment level and average it across all the documents containingclass i ie cwAcci =

sumd pwAccdi

Ni Note that our approach

for computing class-wise scores prevents documents with arelatively larger number of class instances from dominating thescore and in this sense differs from existing approaches [25]

V RESULTS

We report quantitative results using the measures describedin Section IV-C Table IV reports Average Precision and TableV reports class-wise average IOUs and per-pixel accuraciesQualitative results can be viewed in Figure 4 Despite thechallenges posed by manuscripts our model performs rea-sonably well across a variety of classes As the qualitativeresults indicate the model predicts accurate masks for almostall the regions The results also indicate that our model handlesoverlap between Holes and Character line segments wellFrom ablative experiments we found that our choice of focalloss was crucial in obtaining accurate mask boundaries Unliketraditional semantic segmentation which would have produceda single blob-like region for line segments our instance-basedapproach isolates each text line separately Additionally theclear demarcation between Page-Boundary and background in-dicates that our system identifies semantically relevant regionsfor downstream analysis As the result at the bottom of Figure4 shows our system can even handle images with multiplepages thus removing the need for any pre-processing relatedto isolation of individual pages

From quantitative results we observe that Holes Characterline segments Page boundary and Pictures are parsed thebest while Physical degradations are difficult to parse dueto the relatively small footprint and inconsistent patterns indegradations The results show that performance for Penn

in Hand (PIH) documents is better compared to BHOOMImanuscripts We conjecture that the presence of closely spacedand unevenly written lines in latter is the cause In ourapproach two (or more) objects may share the same boundingbox in terms of overlap and it is not possible to determinewhich box to choose during mask prediction Consequentlyan underlying linersquos boundary may either end up not beingdetected or the predicted mask might be poorly localizedHowever this is not a systemic problem since our modelachieves good performance even for very dense BHOOMIdocument line layouts

VI CONCLUSION

Via this paper we propose Indiscapes the first dataset withlayout annotations for historical Indic manuscripts We believethat the availability of layout annotations will play a crucialrole in reducing the overall complexity for OCR and othertasks such as word-spotting style-and-content based retrievalIn the long-term we intend to expand the dataset not onlynumerically but also in terms of layout script and languagediversity As a significant contribution we have also adapted adeep-network based instance segmentation framework custommodified for fully automatic layout parsing Given the generalnature of our framework advances in instance segmentationapproaches can be leveraged thereby improving performanceover time Our proposed web-based annotator system althoughdesigned for Indic manuscripts is flexible and could be reusedfor similar manuscripts from Asian subcontinent We intend toexpand the capabilities of our annotator system in many usefulways For instance the layout estimated by our deep-networkcould be provided to annotators for correction thus reduc-ing annotation efforts Finally we plan to have our datasetinstance segmentation system and annotator system publiclyavailable This would enable large-scale data collection andautomated analysis efforts for Indic as well as other historicalAsian manuscripts The repositories related to the systemspresented in this paper and the Indiscapes dataset can beaccessed at httpsihdiaiiitacin

ACKNOWLEDGMENT

We would like to thank Dr Sai Susarla for enabling accessto the Bhoomi document collection We also thank PoreddyMourya Kumar Reddy Gollapudi Sai Vamsi Krishna for theircontributions related to dashboard and various annotators fortheir labelling efforts

REFERENCES

[1] C Reul M Dittrich and M Gruner ldquoCase study of a highly automatedlayout analysis and ocr of an incunabulumrsquoder heiligen lebenrsquo(1488)rdquoin Proc 2nd Intl Conf on Digital Access to Textual Cultural HeritageACM 2017 pp 155ndash160 1

[2] U Springmann and A Luedeling ldquoOcr of historical printings with anapplication to building diachronic corpora A case study using the ridgesherbal corpusrdquo Digital Humanities Quarterly no 2 2017 1

[3] F Simistira M Seuret N Eichenberger A Garz M Liwicki and R In-gold ldquoDiva-hisdb A precisely annotated large dataset of challengingmedieval manuscriptsrdquo in ICFHR IEEE 2016 pp 471ndash476 1

original size using bilinear interpolation On these generatedmasks NMS with a threshold value of 05 is applied to obtainthe final set of predicted masks

C Evaluation

For quantitative evaluation we compute Average Precision(AP) for a particular IoU threshold a measure widely reportedin instance segmentation literature [44] [48] We specificallyreport AP50 and AP75 corresponding to AP at IoU thresholds50 and 75 respectively [41] In addition we report an overallscore by averaging AP at different IoU thresholds ranging from05 to 095 in steps of 005

The AP measure characterizes performance at documentlevel To characterize performance for each region type wereport two additional measures [25] ndash average class-wise IoU(cwIoU) and average class-wise per-pixel accuracy (cwAcc)Consider a fixed test document k Suppose there are ri regionsof class i and let IoUr denote the IoU score for one suchregion r ie 1 6 r 6 ri The per-class IoU score for class iand document k is computed as cwIoUd

i =sum

r IoUr

ri Suppose

there are Ni documents containing at least a single region ofclass i in ground-truth The overall per-class IoU score forclass i is computed as cwIoU i =

sumd cwIoUd

i

Ni In a similar

manner we define class-wise pixel accuracy pwAccdi at docu-ment level and average it across all the documents containingclass i ie cwAcci =

sumd pwAccdi

Ni Note that our approach

for computing class-wise scores prevents documents with arelatively larger number of class instances from dominating thescore and in this sense differs from existing approaches [25]

V RESULTS

We report quantitative results using the measures describedin Section IV-C Table IV reports Average Precision and TableV reports class-wise average IOUs and per-pixel accuraciesQualitative results can be viewed in Figure 4 Despite thechallenges posed by manuscripts our model performs rea-sonably well across a variety of classes As the qualitativeresults indicate the model predicts accurate masks for almostall the regions The results also indicate that our model handlesoverlap between Holes and Character line segments wellFrom ablative experiments we found that our choice of focalloss was crucial in obtaining accurate mask boundaries Unliketraditional semantic segmentation which would have produceda single blob-like region for line segments our instance-basedapproach isolates each text line separately Additionally theclear demarcation between Page-Boundary and background in-dicates that our system identifies semantically relevant regionsfor downstream analysis As the result at the bottom of Figure4 shows our system can even handle images with multiplepages thus removing the need for any pre-processing relatedto isolation of individual pages

From quantitative results we observe that Holes Characterline segments Page boundary and Pictures are parsed thebest while Physical degradations are difficult to parse dueto the relatively small footprint and inconsistent patterns indegradations The results show that performance for Penn

in Hand (PIH) documents is better compared to BHOOMImanuscripts We conjecture that the presence of closely spacedand unevenly written lines in latter is the cause In ourapproach two (or more) objects may share the same boundingbox in terms of overlap and it is not possible to determinewhich box to choose during mask prediction Consequentlyan underlying linersquos boundary may either end up not beingdetected or the predicted mask might be poorly localizedHowever this is not a systemic problem since our modelachieves good performance even for very dense BHOOMIdocument line layouts

VI CONCLUSION

Via this paper we propose Indiscapes the first dataset withlayout annotations for historical Indic manuscripts We believethat the availability of layout annotations will play a crucialrole in reducing the overall complexity for OCR and othertasks such as word-spotting style-and-content based retrievalIn the long-term we intend to expand the dataset not onlynumerically but also in terms of layout script and languagediversity As a significant contribution we have also adapted adeep-network based instance segmentation framework custommodified for fully automatic layout parsing Given the generalnature of our framework advances in instance segmentationapproaches can be leveraged thereby improving performanceover time Our proposed web-based annotator system althoughdesigned for Indic manuscripts is flexible and could be reusedfor similar manuscripts from Asian subcontinent We intend toexpand the capabilities of our annotator system in many usefulways For instance the layout estimated by our deep-networkcould be provided to annotators for correction thus reduc-ing annotation efforts Finally we plan to have our datasetinstance segmentation system and annotator system publiclyavailable This would enable large-scale data collection andautomated analysis efforts for Indic as well as other historicalAsian manuscripts The repositories related to the systemspresented in this paper and the Indiscapes dataset can beaccessed at httpsihdiaiiitacin

ACKNOWLEDGMENT

We would like to thank Dr Sai Susarla for enabling accessto the Bhoomi document collection We also thank PoreddyMourya Kumar Reddy Gollapudi Sai Vamsi Krishna for theircontributions related to dashboard and various annotators fortheir labelling efforts

REFERENCES

[1] C Reul M Dittrich and M Gruner ldquoCase study of a highly automatedlayout analysis and ocr of an incunabulumrsquoder heiligen lebenrsquo(1488)rdquoin Proc 2nd Intl Conf on Digital Access to Textual Cultural HeritageACM 2017 pp 155ndash160 1

[2] U Springmann and A Luedeling ldquoOcr of historical printings with anapplication to building diachronic corpora A case study using the ridgesherbal corpusrdquo Digital Humanities Quarterly no 2 2017 1

[3] F Simistira M Seuret N Eichenberger A Garz M Liwicki and R In-gold ldquoDiva-hisdb A precisely annotated large dataset of challengingmedieval manuscriptsrdquo in ICFHR IEEE 2016 pp 471ndash476 1

[4] A Pappo-Toledano F Chen G Latif and L Alzubaidi ldquoAdoptivethresholding and geometric features based physical layout analysis ofscanned arabic booksrdquo 2018 IEEE 2nd Intl Workshop on Arabic andDerived Script Analysis and Recognition (ASAR) pp 171ndash176 2018 1

[5] M W A Kesiman J-C Burie G N M A Wibawantara I M GSunarya and J-M Ogier ldquoAmadi lontarset The first handwrittenbalinese palm leaf manuscripts datasetrdquo in ICFHR IEEE 2016 pp168ndash173 1

[6] K Chen M Seuret M Liwicki J Hennebert and R Ingold ldquoPagesegmentation of historical document images with convolutional autoen-codersrdquo in ICDAR IEEE 2015 pp 1011ndash1015 1

[7] J Sahoo ldquoA selective review of scholarly communications on palm leafmanuscriptsrdquo Library Philosophy and Practice (e-journal) 2016 1

[8] Y B Rachman ldquoPalm leaf manuscripts from royal surakarta indonesiaDeterioration phenomena and care practicesrdquo Intl Journal for thePreservation of Library and Archival Material vol 39 no 4 pp 235ndash247 2018 1

[9] D U Kumar G Sreekumar and U Athvankar ldquoTraditional writingsystem in southern indiapalm leaf manuscriptsrdquo Design Thoughts vol 92009 1

[10] D Valy M Verleysen S Chhun and J-C Burie ldquoA new khmer palmleaf manuscript dataset for document analysis and recognition Sleukrithsetrdquo in Proc of the 4th Intl Workshop on Historical Document Imagingand Processing ACM 2017 pp 1ndash6 1

[11] J A Sanchez V Bosch V Romero K Depuydt and J De DoesldquoHandwritten text recognition for historical documents in the transcrip-torium projectrdquo in Proc of the First Intl Conf on Digital Access toTextual Cultural Heritage ACM 2014 pp 111ndash117 1

[12] T M Rath and R Manmatha ldquoWord spotting for historical documentsrdquoIJDAR vol 9 no 2-4 pp 139ndash152 2007 1

[13] M Kassis A Abdalhaleem A Droby R Alaasam and J El-SanaldquoVml-hd The historical arabic documents dataset for recognition sys-temsrdquo in 1st Intl Workshop on Arabic Script Analysis and RecognitionIEEE 2017 1

[14] M Suryani E Paulus S Hadi U A Darsa and J-C Burie ldquoThehandwritten sundanese palm leaf manuscript dataset from 15th centuryrdquoin ICDAR IEEE 2017 pp 796ndash800 1 2

[15] C Clausner A Antonacopoulos T Derrick and S Pletschacher ldquoIc-dar2017 competition on recognition of early indian printed documents-reid2017rdquo in ICDAR vol 1 IEEE 2017 pp 1411ndash1416 2

[16] C K Savitha and P J Antony ldquoMachine learning approaches forrecognition of offline tulu handwritten scriptsrdquo Journal of PhysicsConference Series vol 1142 p 012005 nov 2018 2

[17] A Abeysinghe and A Abeysinghe ldquoUse of neural networks in archae-ology preservation of assamese manuscriptsrdquo International Seminaron Assamese Culture amp Heritage 2018 2

[18] P N Sastry T V Lakshmi N K Rao and K RamaKrishnanldquoA 3d approach for palm leaf character recognition using histogramcomputation and distance profile featuresrdquo in Proc 5th Intl Conf onFrontiers in Intelligent Computing Theory and Applications Springer2017 pp 387ndash395 2

[19] N S Panyam V L TR R Krishnan and K R NV ldquoModelingof palm leaf character recognition system using transform based tech-niquesrdquo Pattern Recogn Lett vol 84 no C Dec 2016 2

[20] Z Shi S Setlur and V Govindaraju ldquoDigital enhancement of palm leafmanuscript images using normalization techniquesrdquo in 5th Intl Conf OnKnowledge Based Computer Systems 2004 pp 19ndash22 2

[21] D Sudarsan P Vijayakumar S Biju S Sanu and S K ShivadasldquoDigitalization of malayalam palmleaf manuscripts based on contrast-based adaptive binarization and convolutional neural networksrdquo in IntlConf on Wireless Communications Signal Processing and Networking(WiSPNET) 2018 2

[22] C Wick and F Puppe ldquoFully convolutional neural networks for pagesegmentation of historical document imagesrdquo in DAS IEEE 2018 pp287ndash292 2

[23] H Wei M Seuret K Chen A Fischer M Liwicki and R IngoldldquoSelecting autoencoder features for layout analysis of historical docu-mentsrdquo in Proc 3rd Intl Workshop on Historical Document Imagingand Processing ser HIP rsquo15 ACM 2015 pp 55ndash62 2

[24] S S Bukhari T M Breuel A Asi and J El-Sana ldquoLayout analysis forarabic historical document images using machine learningrdquo in ICFHR2012 IEEE 2012 pp 639ndash644 2

[25] K Chen M Seuret J Hennebert and R Ingold ldquoConvolutional neuralnetworks for page segmentation of historical document imagesrdquo inICDAR vol 1 IEEE 2017 pp 965ndash970 2 7

[26] B Barakat A Droby M Kassis and J El-Sana ldquoText line segmentationfor challenging handwritten document images using fully convolutionalnetworkrdquo in ICFHR IEEE 2018 pp 374ndash379 2

[27] M W A Kesiman D Valy J Burie E Paulus M Suryani S HadiM Verleysen S Chhun and J Ogier ldquoICFHR 2018 competition on doc-ument image analysis tasks for southeast asian palm leaf manuscriptsrdquoin ICFHR 2018 pp 483ndash488 2

[28] Proc 4th Intl Workshop on Historical Document Imaging and Process-ing Kyoto Japan November 10-11 2017 ACM 2017 2

[29] Proc 3rd Intl Wksp on Historical Document Imaging and ProcessingHIPICDAR 2015 ACM 2015 2

[30] R S Sabeenian M E Paramasivam P M Dinesh R Adarsh andG R Kumar ldquoClassification of handwritten tamil characters in palmleaf manuscripts using svm based smart zoning strategiesrdquo in ICBIPACM 2017 2

[31] M W A Kesiman D Valy J-C Burie E Paulus M Suryani S HadiM Verleysen S Chhun and J-M Ogier ldquoBenchmarking of documentimage analysis tasks for palm leaf manuscripts from southeast asiardquoJournal of Imaging vol 4 no 2 p 43 2018 2

[32] D Valy M Verleysen S Chhun and J-C Burie ldquoCharacter and textrecognition of khmer historical palm leaf manuscriptsrdquo in ICFHR 082018 pp 13ndash18 2

[33] E Paulus M Suryani and S Hadi ldquoImproved line segmentation frame-work for sundanese old manuscriptsrdquo Journal of Physics ConferenceSeries vol 978 p 012001 mar 2018 2

[34] D Doermann E Zotkina and H Li ldquoGEDI-a groundtruthing environ-ment for document imagesrdquo in Ninth IAPR Intl Workshop on DocumentAnalysis Systems 2010 2

[35] A Garz M Seuret F Simistira A Fischer and R Ingold ldquoCreatingground truth for historical manuscripts with document graphs andscribbling interactionrdquo in DAS IEEE 2016 pp 126ndash131 2

[36] C Clausner S Pletschacher and A Antonacopoulos ldquoAletheia-an ad-vanced document layout and text ground-truthing system for productionenvironmentsrdquo in ICDAR IEEE 2011 pp 48ndash52 2

[37] ldquoWeb aletheiardquo [Online] Available httpsgithubcomPRImA-Research-Labprima-gwt-lib 2

[38] M Wursch R Ingold and M Liwicki ldquoDivaservicesa restful webservice for document image analysis methodsrdquo Digital Scholarship inthe Humanities vol 32 no 1 pp i150ndashi156 2016 2

[39] B Gatos G Louloudis T Causer K Grint V Romero J A SanchezA H Toselli and E Vidal ldquoGround-truth production in the transcrip-torium projectrdquo in DAS IEEE 2014 pp 237ndash241 2

[40] ldquoPenn in hand Selected manuscriptsrdquo httpdlalibraryupennedudlamedrensearchhtmlfq=collection facetrdquoIndicManuscriptsrdquo 2

[41] K He G Gkioxari P Dollar and R B Girshick ldquoMask r-cnnrdquo ICCVpp 2980ndash2988 2017 4 7

[42] T Lin P Dollar R B Girshick K He B Hariharan and S J BelongieldquoFeature pyramid networks for object detectionrdquo in CVPR 2017 pp936ndash944 4

[43] K He X Zhang S Ren and J Sun ldquoDeep residual learning for imagerecognitionrdquo in CVPR 2016 pp 770ndash778 4

[44] T Lin M Maire S J Belongie L D Bourdev R B GirshickJ Hays P Perona D Ramanan P Dollar and C L Zitnick ldquoMicrosoftCOCO common objects in contextrdquo CoRR vol abs14050312 2014[Online] Available httparxivorgabs14050312 4 7

[45] J Deng W Dong R Socher L-J Li K Li and L Fei-Fei ldquoImagenetA large-scale hierarchical image databaserdquo in CVPR IEEE 2009 pp248ndash255 4

[46] S Ren K He R Girshick and J Sun ldquoFaster r-cnn Towards real-time object detection with region proposal networksrdquo in NIPS 2015pp 91ndash99 5

[47] T-Y Lin P Goyal R Girshick K He and P Dollar ldquoFocal loss fordense object detectionrdquo in ICCV 2017 pp 2980ndash2988 5

[48] M Cordts M Omran S Ramos T Rehfeld M EnzweilerR Benenson U Franke S Roth and B Schiele ldquoThe cityscapes datasetfor semantic urban scene understandingrdquo CoRR vol abs160401685

2016 [Online] Available httparxivorgabs160401685 7