A Cost Efficient Approach to Correct OCR Errors in Large Document Collections

Deepayan Das, Jerin Philip, Minesh Mathew and C. V. JawaharCenter for Visual Information Technology, IIIT Hyderabad, India.

{jerin.philip, deepayan.das, minesh.mathew}@research.iiit.ac.in, jawahar@iiit.ac.in

Abstract—Word error rate of an OCR is often higher thanits character error rate. This is specially true when OCRs aredesigned by recognizing characters. High word accuracies arecritical to tasks like creation of content in digital librariesand text-to-speech applications. In order to detect and correctthe misrecognised words, it is common for an OCR moduleto employ a post-processor to further improve the wordaccuracy. However, conventional approaches to post-processinglike looking up a dictionary or using a statistical languagemodel (SLM), are still limited. In many such scenarios, it isoften required to remove the outstanding errors manually.

We observe that the traditional post processing schemes lookat error words sequentially, since OCRs process documents oneat a time. We propose a cost efficient model to address the errorwords in batches rather than correcting them individually. Weexploit the fact that a collection of documents, unlike a singledocument, has a structure leading to repetition of words. Suchwords, if efficiently grouped together and corrected as a wholecan lead to significant reduction in the cost. Correction canbe fully automatic or with a human in the loop. Towardsthis we employ a novel clustering scheme to obtain fairlyhomogeneous clusters. We compare the performance of ourmodel with various baseline approaches including the casewhere all the errors are removed by a human. We demonstratethe efficacy of our solution empirically by reporting more than70% reduction in the human effort with near perfect errorcorrection. We validate our method on Books from multiplelanguages.

Keywords-OCR, Batch Correction, Clustering, Post-Processing

I. INTRODUCTION

The past decade witnessed a growing interest towards thecreation of huge digital libraries by digitizing books [1, 2].One of the crucial steps towards digitization involves therecognition and reconstruction of document image collec-tion(s) using an OCR. The recognition module in the contextof digitizing collections of books could be considerably dif-ferent from that of recognizing a single document image [3].In this work, we extend this idea to error correction indocument image collections.

Often the recognition module of the OCRs have an au-tomatic error correction module embedded. This may beusing a dictionary or a statistical language model (SLM).However, many applications need further improvement inaccuracy. This demands a human intervention for removingthese errors. In this paper, we propose enhancements to thenaive human correction approach which reduces the cost forhuman expert review by more than 70%. Our work is guided

Figure 1: The proposed pipeline for batch correction processwhere the error instances are clustered and corrected in one go. Fora group of error instances, the correct label is chosen and applied.The correct label can be either chosen by a human annotator (a)or generated automatically (b).

by the following two insights. First - the OCR module makeserrors consistently. For two word images drawn from thesame type of document, similar noise leads to the same kindof errors. We demonstrate this in Figure 3 where instancesof same word images drawn from a document collection aremisclassified consistently by the OCR. The second, there canonly be a finite vocabulary for a book and majority of wordsunknown to the error detection system which may includenamed entities and domain specific terms repeat themselvesthroughout the collection. This is further validated in Figure2 where we show that a subset of words in collection occurvery frequently and constitutes almost 50% of the totalwords present. Under this setting, grouping based on imagefeatures or similarity in the predictions of the OCR canprovide cues for automatic correction or aide a human editor.We model the problem of error correction as batch correctionwhere the human reviewer reviews and corrects errors inbatches. Figure 1 presents an overview of our proposed batchcorrection scheme. Word image-prediction pairs extractedfrom a collection of documents form groups based on theirimage and text similarity. In case such a group is recognizedincorrectly by the OCR, only one instance from the groupneeds to be corrected which is then propagated to the restof the group elements. Thus, correction needs to be made

arX

iv:1

905.

1173

9v1

[cs

.CV

] 2

8 M

ay 2

019

only once which reduces the cost of correction drastically.The correction can either be made with the help of a humaneditor or else the correction process can be automated. Wediscuss both kinds of batch correction processes in detaillater in this paper. The major contributions of this work are:• We demonstrate how clustering can induce an au-

tomatic correction and reduce the manual effort incorrection significantly.

• We successfully demonstrate ability to scale the clus-tering scheme to large collection of 100 books.

A. Related Work

Conventional approaches to error detection and correctionreduces to finding the closest match for an invalid word ina known vocabulary [4, 5]. Bassil and Alwani [5] put forthone of the first works which explored in detail OCR post-processing methods, in which they consider three modes ofcorrection. In the simplest of approaches, corrections couldbe performed manually by a human proofreader. Next, adictionary-based method similar to what modern day wordprocessors are equipped with was proposed. A possible cor-rection is suggested once an error word was detected. Thisis accomplished by finding a word in the dictionary withminimum edit distance to the error word which becomesthe correction proposal. Dictionary-based approaches couldnot capture errors in the grammar where words were correctaccording to the dictionary, but not in the surroundingcontext. Ability to correct such mismatches was broughtabout by grammar-aware models like Statistical LanguageModels using larger language context [6, 7]. SLMs don’twork well for many languages which lack corpus to trainon. Also, they run into issues when newer out-of trainingdomain words come in books. Further, Smith [8] indicatesunless carefully applied, a language model can do more harmthan good. Hence it becomes necessary to review the resultsof a conventional OCR system bringing a human in the loopfor the perfect digital reproduction of a book. To involvehuman in the loop, projects as early as Project Gutenberg[1] introduced Distributed Proofreading [9] approaches. Twoproofreaders, having access to a book’s page images, refineits OCR outputs in turns. A demerit, in this case, is thatthe entire book has to be visited for proofreading. Von Ahnet al. [10] using ReCaptcha reports use of crowd-sourcing totranscribe word images where the OCR outputs are detectedto be erroneous. While the corrections are made only in thecase of suspected errors, the efforts ignore the possibilityof grouping similar misrecognized images and propagatingthe correct label to each instance in one go. Observing OCRerrors to be highly correlated, Abdulkader and Casey [11]proposes a low-cost method to improve the required human-hours needed for correction using clustering. They groupby OCR outputs first, followed by finding subgroups usingthe word-image similarities. The above approach however,assumes the clusters to be completely homogeneous and thus

fail to address cases where the clusters might contain morethan one label.

Figure 2: The frequency of unique words in a collection ofdocuments. A subset of words in the collection vocabulary have avery high frequency and accounting for 50% of the words presentin the collection. Thus it is safe to assume that if errors occurringin this subset are grouped and corrected in a batch, it can lead toa significant reduction in correction cost.

In our next step, we review massive-digitization effortsin the past. Initiatives for a digital library for books throughlarge scale digitization in the past include Project Gutenburg[1], Google Books [2]. One of the main objectives of suchprojects is to provide content level access (enable search andretrieval) over the entire digitized collection.

Baird et al. [12], Taghva et al. [13] note that enablinginformation retrieval in such databases is hampered by errorsin OCR outputs. Past works turn to humans for correctingthe last array of errors left in the pipeline post recognition[10, 11]. All these leave scope for improvement in thespace of error correction, especially addressing challengeswhile scaling up the number of books. Our work is alsomotivated by the works of Abdulkader and Casey [11]. Wegroup the errored predictions based on their image and textsimilarity and present them to a human editor. The humaneditor then decides the label for the cluster and also thecomponents (elements present in a cluster) to which the labelshall be assigned to. The instances where the cluster labeldo not match the content of the word image are addressedseparately by the editor. This mitigates the propagation oferrors for clusters that are not homogeneous.

II. COST EFFECTIVE CORRECTION

In this section we formulate the problem of error correc-tion and propose two strategies for using our batch correctionmethod to address this issue.

A. Problem Formulation

Recognition modules of OCR systems operate at a charac-ter or word level resulting in transcribing word-images into

Figure 3: Consistent errors generated by OCR for a given document collection. Each row represents different images for the same wordand it’s corresponding OCR prediction in the green text box. We can observe that for similar degradations, the OCR outputs similar errorpatterns.

a textual string. Errors in such a setup are inevitable andthe cost of manual correction is significantly high. Since itis practically impossible to verify each word manually, wepropose to have an independent error detection mechanismoperating on the OCR predictions. Assuming that such asystem has a low False Negative Rate, only instances wherethe OCR prediction is not agreed upon by the error detectionpipeline which we denote hereafter as error instances, wouldthen need to be corrected. We assume that the errors aredetected with a dictionary or an appropriate error detectionmodule. Our contribution is to make further improvements tothis setup by observing that an OCR based system is proneto make systematic errors. Due to the nature of learning,multiple instances of the same word could be misclassifiedto the same wrong label. We propose a grouping of suchmisclassifications in a collection of documents which enablecorrecting these multiple errors in one go. In this work, weuse a word-level OCR and a dictionary for the error detectionmodule.

One can categorize the agreement between the recognitionmodule and the error detector into four:

1) Error False Positives (EFP): Words that are falselyflagged as error by the detection module since theydo not exist in the dictionary OOV.

2) Error True Positives (ETP): Errors of the OCR whichare correctly detected by the error detection module.

3) Recognizer False Negatives (RFN): Words exist in thedictionary but are not the correct transcriptions of theword image.

4) True Negatives (TN) of the error detection module:Recognizer correctly predicts word image, and thedetection module is in agreement.

As far as the error correction is concerned, we wouldlike to take human help or automatically correct the wordscategorized as ETP. Note that the words in TN after error

detection are correct words and nothing needs to be done.The words in RFN cannot be detected as an error in isolation.Their correction needs larger language context and is out ofscope for this paper.

We propose a cost based evaluation to demonstrate theefficacy of our method. To this end, we first enumerate allpossible edit actions a human in the loop has available andassociate a cost with each action. We define a verificationcost Cv for the case where the reviewer just has to verifyan already correct prediction to be a valid word. We defineaverage word typing cost Ct for cases where correctionshave to be fully typed out. For cases where a dictionaryprovides correction proposals in a drop-down fashion, wedefine a cost Cd.

For a naive correction process (process where no batchingis involved), the editor will have to type out corrections inETP and verify a word wrongly classified in EFP. The totalcost involved turns out to be C1 = (|ETP|)Ct + (|EFP|)Cv .We denote this method hereafter as Typing. If for someerror instances, the editor has an additional option to selectfrom a set of correction proposals, the cost reduces toC2 = |ETPt|Ct + |ETPd|Cd + |EFP|Cv such that ETPt, ETPdforms a partition of ETP. Here ETPt refers to the error truepositives which can only be corrected via Typing whereasETPd refers to the error true positives that can be correctedby choosing the correct suggestion from the set of correctionproposals. (Here, |X| denotes the cardinality of set X .) Thismethod is denoted as Typing+Selection hereafter.

We hypothesize correcting similar instances in EFP andETP together can make digitization efforts more efficient.As mentioned above, we propose an approach that groupstogether error instances based on some similarity metric andpropagate the correction of one of these to the rest of thegroup. We emphasize selection of the correction candidatefor a group of error words can be either fully automated or

Figure 4: Pipeline of proposed batch correction approach. Given word images (wi. . .wn) and its corresponding OCR predictions (li. . . ln)we form clusters. Next, the clusters containing error instances are sent for correction. We employ two forms of correction approacheswhich are shown in (a) when the human editor decides the label for a cluster and in (b) when the cluster label is generated automatically.

done with human aid. We discuss both the propositions indetail later in this section. In the ideal case, word imageswith same ground truth will be grouped together, and theability to correct them in one go would provide an efficientway for humans in the loop to correct large documentcollections. If we could group the error instances based ontheir ground truths as C1, C2 . . . C|V |, each of these groupscould be corrected in just one action from editor leading toa cost of VtCt+VdCd+VvCv such that Vt+Vd+Vv = |V |.Here Vt, Vd and Vv are numbers of clusters requiring typing,selection from dictionary and verification respectively.

B. Correction ApproachOur proposed model for error correction is presented in

Figure 4. The document images, segmented at word level gothrough the OCR pipeline which assigns them labels. Theword images and their corresponding predictions are sub-sequently sent through a clustering pipeline, which groupsthe word images based on their image and text similarity.We discuss the clustering pipeline along with the featureson which the clustering is performed in Section III. Nextwe perform an error detection on the components of eachcluster and identify those clusters in which error instancesoccur. Only those clusters which contain error instances aresent for either of the two correction techniques- automatedor human aided which are discussed below.

Automated approach: For a given cluster containingword images and their corresponding OCR predictions, themost frequent prediction is chosen to be the representative ofthe whole cluster and its label is propagated to the remainingcluster elements. Two scenarios arise out of such a setting.For a given cluster-

1) The number of correct predictions is more than incor-rect predictions.

2) The number of incorrect predictions is more than thenumber of correct predictions.

In the first case, words appearing in ETP get correctedautomatically without any further manual corrective actionother than verification. In the second case, words appearingin EFP (proper nouns, acronyms, technical keyword, etc.) getcorrected without much cost, while for clusters containingETP, even the correct predictions end up being assignedthe wrong label. Thus a human editor is required to verifythe assigned label with the actual word image for everyerroneous prediction and make keyboard entries wherevernecessary. This leads to an added correction cost.

Human aided approach: We allow a human editor topick the representative of the cluster. This reduces the costby eliminating the chances of error propagation which arisewhen labels are generated automatically. However, this alsomandates that a human editor be present throughout thecorrection process. In case of ETP, the editor can enterthe correction once and the correction is propagated to allmatching images. Our method here reduces the cognitiveload for the human, thereby improving efficiency.

In the above two approaches we consider the clusters tobe completely homogeneous. Clusters containing impuritiesand the relevant correction approach is discussed later in thepaper.

III. GROUPING ERROR WORDS

In this section, we provide the details of our approachfor grouping error words together. As discussed earlier, thissignificantly reduces human cost.

A. Features for Clustering

For every error instance, we have two types of featuresfor use in clustering: text predictions of OCR and features

from word-image.Image Features: We use the pre-final layer repre-

sentations from deep neural networks trained to classifyword-images. Such representations capture the discrimina-tory information between different word-images and havedemonstrated success in embedding similar images together[14]. The activation for an image can be considered as acompact representation in a continuous space. For clusteringthe above features, we employ the k-means [15] algorithm.

Text Features: For text features we propose using theword predictions of the OCR. A natural distance measure forsuch features is the edit distance, which has been found to beof significant help for error detection in past work. However,approaches like k-means are ill-suited to the discrete natureof these features and our distance measure. Therefore, wepropose using a Minimum Spanning Tree (MST) basedapproach [15] using pairwise edit-distance to cluster variantsof text predictions. This could also group consistent errorswhich comprise of error instances where the (1) Predictionis right but error detection is in disagreement. (2) Wherefor the same kind of word-image OCR consistently give thesame erroneous prediction due to bias in training data.

Image Features and Text: Word images with high visualsimilarity but having different text content can be groupedinto the same cluster since they might be close to each otherin the image feature space. This leads to fragmentation orformation of impure cluster. Assuming one true label percluster can induce an additional cost of correcting wordinstances whose ground truth is different from the assignedlabel. To address the intra-cluster variability we furtherpartition each cluster into sub-clusters by leveraging thetextual transcription of each word image such that wordsthat lie within a predefined edit distance, can be groupedinto the same sub-cluster.

B. Clustering Algorithms

In a simpler first approach over a fewer number of books,we use k-means and MST based clustering algorithm togroup error instances together. While the two algorithmswork well for a fewer number of books, they are not wellsuited to scale to a larger setting. We address this by usinga Locality Sensitive Hashing (LSH) based nearest neighbourcomputation [16] in our clustering pipeline. We discuss indetail the algorithms and their suitability below.

We employ k-means on the image representations withnumber of clusters (k) set to number of unique words in acollection. The k-means algorithm has a time complexity ofO(n2) where n is the number of error instances detected byour pipeline.

We use MST clustering on the text predictions to fur-ther partition the clusters. We consider the predictions asvertices of a weighted undirected graph, and the pairwiseedit-distance between two vertices form the edge weights.Distances between vertices are scaled to [0, 1]. A MST is

constructed and edges with weights greater than a thresholdare discarded, which results in a forest where each connectedcomponent forms a cluster.

Degradations in print, paper or both over time are preva-lent in older documents. Font styles and variations differentfrom the OCR’s training distribution used by a commonpublishing system across these books could be similar inthe image space. Similar noise in the images like the cutsand merges lead to consistent errors in OCR. This priordomain knowledge can be incorporated and taken advantageof while clustering. Under these circumstances, we find LSHwell suited for scaling up correction in our problem setting.LSH tends to approximate the nearest neighbour search in away such that items which are similar are hashed into thesame ‘bucket’. Consistency in noise leads to similar hashesfor features from images with similar content. Search spaceis now limited to the bucket of word-images for which hashmatches the query image. This makes the process ordersfaster.

IV. DATASET AND EVALUATION PROTOCOLS

Our dataset comprises of books that have been fullydigitized by our OCR module. They are categorized into twotypes. The first is a smaller subset having books that havebeen verified by a human expert, while the second composesa larger subset containing unverified books. We denote theformer as fully annotated data and the latter as partiallyannotated data. We seek to evaluate both these datasets ontwo separate objectives.

For the fully annotated dataset, our objective is to findwhich among the proposed clustering approaches works bestfor document collection. For the partially annotated dataset,we look to evaluate the scalability of the proposed clusteringapproaches on larger unverified data. Table I gives details ofour dataset, both annotated and partially annotated used inour experiments and the evaluation methods directed towardsascertaining what works for each objective.

scale language #books #pages #words # unique

FA English 15 2417 0.73M 30KHindi 32 4287 1.20M 63K

PA Hindi- annotated 50 200 30K 6K- unannotated 100 25K 5M* 80K*

Table I: Details of the books used in our work. Here FArefers to the fully annotated books whereas PA refers to thepartially annotated books.

A. Annotated Data

The annotated dataset comprises of 19 books from En-glish and 32 books from Hindi. Pages from the books aresegmented at a word level and annotated by human experts.5 books are set aside from each of the languages to train theOCR while rest of the books are used for testing and furtherbatch correction experiments.

English HindiMethod Automated Human Automated Human

Typing (Typing + Selection) Typing (Typing + Selection) Typing (Typing + Selection) Typing (Typing + Selection)- Static Growing - Static Growing - Static Growing - Static Growing

k-means(I) 1.130 0.873 0.692 0.689 0.527 0.372 1.013 0.714 0.648 0.494 0.366 0.234LSH(I) 0.939 0.732 0.695 0.283 0.232 0.222 0.944 0.664 0.659 0.162 0.135 0.134MST(T) 1.000 0.740 0.695 0.199 0.187 0.187 1.000 0.695 0.681 0.142 0.133 0.132k-means(I) + MST(T) 1.000 0.853 0.653 0.607 0.459 0.327 0.960 0.681 0.634 0.281 0.217 0.191LSH(I) + MST(T) 0.947 0.739 0.689 0.285 0.232 0.222 0.949 0.666 0.651 0.153 0.129 0.128

Table II: Evaluation of costs of each approach proposed in this paper. The numbers reported are relative to Full Typingmethod. We observe a decrease in cost as we go left to right for each clustering approach for books of a given language.‘I’ stands for image features, and ‘T’ stands for prediction text.

B. Partially Annotated Data

In order to demonstrate the scalability of our approach,we run our experiments on a larger collection containing 100Hindi books. Most of these books were printed decades ago,resulting in degradation in quality of pages. The collectionconsists of almost 25,000 pages with more than 5 millionwords. A subset of 200 pages across 50 books are set asideas test set, for which we obtain bounding boxes and groundtruths annotated by human experts.

C. Evaluation on Fully Annotated Dataset

We want to find how many time units would be saved for ahuman editor using our pipeline compared to the case whereeach error instance have to be visited individually. The costis measured in units of seconds of human effort put intocorrection. The following values are used for computing thecost in simulations. For verification cost Cv , we supply a 1second and for picking a choice from suggestions, we setcost Cd to be 5 seconds. The cost of typing, Ct, is set to 15seconds, for each word.

The numbers are compared across all proposed clusteringapproaches. Having fully annotated ground-truth informationgives complete cost required in this setting.

D. Evaluation on Partially Annotated Dataset

Here, we evaluate the performance of our approach ona large collection of 25,000 pages. We estimate the per-formance on this collection by explicitly measuring theperformance on the test subset of 200 pages. Only thistest set is used to infer performance, even though we runclustering on a larger set varying the size ranging from 200to 25,000 pages. Please note that the performance reportedin this collection is only an approximation.

We hypothesize that the increase in word accuracy trans-lates to a reduction in correction cost. Also, since the subsetof pages used in evaluation belong to the same pool ofbooks which the larger clustering algorithm is run on, it isreasonable to assume that decrease in cost during evaluationis indicative of a decrease in the larger set of pages.

V. EXPERIMENTS, RESULTS AND DISCUSSION

In this section we describe the various components ofour proposed batch correction model. We briefly discuss

adapting our cost formulation to account for impuritieswhen clusters are not homogeneous. Results for both batchcorrection schemes on annotated data, followed by erroranalysis of our clustering algorithms is then presented.Finally, we illustrate the performance of our model on thelarge scale, partially annotated dataset.

A. Our Pipeline

Our setup consists of the following components - an OCRmodule for recognizing word images, a Convolutional Neu-ral Network(CNN) for extracting representations from theseword images and an error detection module for verifying theaccuracy of these predictions. Our OCR implementation fol-lows a hybrid architecture with convolutional and recurrentlayers first proposed by Shi et al. [17] in their work towardsscene-text recognition.

We trained two OCRs – one for each language. Fortraining, we set aside 5 books each from English and Hindibook datasets respectively. The English language OCR wastrained on word images from nearly 600 pages (∼160Kwords) while the Hindi language OCR was trained on wordimages from approximately 650 pages (∼180K words).

The CNN based feature extractor used in our experimentsfollows the architecture described in Krishnan and Jawahar[14]. The network was initially trained on synthetic hand-written word images and later fine-tuned on a real-worldcorpus. Real data used in training this network is the sameas 160K word images which were used for training our OCR.The segmented word-images are fed to the network and thepre-final layer activations are used as features for clustering.

The error detection module is realised by a dictionary.An instance is determined to be erroneous if its predictionis not present in the dictionary. To suggest correctionsfor an error instance, the dictionary requires a reasonablygood vocabulary. We generate a base dictionary by usingWikipedia dumps for the respective language. For each bookwhile testing, we enrich the corresponding base dictionary’svocabulary further using ground truths of books used fortraining but not the ones we are testing. We use two variantsof this dictionary - one Static and the other Growing. TheGrowing allows for addition of new words to dictionary, likehow modern word processors do. In our grouped correctionscenario same words could be scattered across clusters and

Figure 5: Qualitative results of k-means + MST clustering on English dataset. Images, relevant to the cluster are marked correct whilethe false positives are crossed out.

Growing dictionary speeds up correction by not having totype again the words already corrected.

B. Cluster Impurity

One of the limitations of clustering algorithms like k-means or MST is their inability to form completely homo-geneous clusters. Despite our efforts in fusing image andtext features together in order to minimize the impurities,still outliers manage to creep into the clusters. This canbe verified in Figure 5. This poses a serious drawback inour error correction pipeline. Up until now we consideredour clusters to be homogeneous and formulated our costaccordingly. However, in practice this can lead to wrong costestimation. For automated approach, cluster impurity canlead to assignment of labels to instances which do not sharethe same ground truth. Thus an annotator needs to revisiteach cluster and correct all unwarranted cluster assignments.

For human in the loop, we let the human assign labelsto the cluster components. A human can correct impureparts of the cluster by visual inspection through Typing orSelection. Consistent errors can be corrected for a group inthis case, unlike in automated approach giving this methodan advantage.

C. Results and Discussions

All costs in this work are computed relative to Typing.Table III delineates the cost for correction without groupingefforts. We experiment with setups involving no dictionary,static as well as growing dictionaries, restricting the editactions available accordingly. We find Typing + Selectionoutperforms Typing and Growing outperforms Static, asexpected.

In Table II, we compare the cost of correction whenwe employ different clustering schemes. Here correctionsare performed in batches. The rows correspond to cluster-ing algorithms. Our results are across the two correction

Typing Typing + Selection- Static Growing

English 1.000 0.740 0.695Hindi 1.000 0.686 0.681

Table III: Relative cost of correction with respect to fulltyping when no batching is involved.

approaches - the first which is automated and the secondinvolving a human editor.

The order among relative costs for edit actions anddictionary variants are consistent with the case withoutbatch correction (Table III). Further, we find sequentialrefinement of clusters using image features and then text-features perform best among different clustering schemas.For the automated approach, k-means on image featuresfollowed by MST on text features achieve the lowest costfor both the languages. When involving human editor in theprocess, for Hindi, LSH on image features and refining withMST works best, while for English data MST on text wordpredictions seems to achieve the lowest cost.

Correction methods involving human editor consistentlyoutperforms the automated correction approach, even withthe former restricted in actions and in dictionary. This can beattributed to the failure of automatic approach in determiningwhich is the correct prediction in a cluster which is largelyimpure.

D. Error Analysis

We discuss failure cases of our proposed correction pro-cess with a few qualitative examples. For text predictionsclustered using MST algorithm, a few error cases are illus-trated in Figure 6a. The recognition module’s high confusionin predicting numbers and punctuation extends to clusteringusing text predictions. But there exists strong cues here inthe image feature space which can be used to group samplesseparately.

Figure 6b shows failures in clustering solely using image

(a) Text predictions (b) Image features

Figure 6: Failure cases for clustering on text and image featuresrespectively. Each row in the above figure represents one cluster.The text predictions are depicted in green text box.

features. Instances containing ‘Carpenter’ and ‘Capulet’ aregrouped into the same cluster although there is a significantdifference between their text predictions. Image featurebased clustering alone fails to obtain a pure cluster here,but text predictions’ similarity can be used to make clustersmore pure. We demonstrate such successful refinement inFigure 5. ‘Capulet’ is one such correction proposal, but theentry corresponding to ‘Carpenter’ is no longer associated.

Failure cases of the combined clustering approach areindicated in Figure 5. Predictions ‘fool’ and ‘food’ are in-herently different, but still managed to be clustered together.This is likely due to these being very near in image and textspace.

E. Results on Large Dataset

Collection Size (# of Words X 1K)

Wor

d A

ccur

acy

(%)

Figure 7: Result on the unannotated data. We observe that asthe number of words in the collection increases the automatedbatch correction method’s ability to correct the errored predictionsimprove which is reflected by the increase in OCR accuracy.

We vary the size of the collection from 200 to 25,000pages and estimate the accuracy on the 200 fully annotatedpages. Due to the nature of the word images, the wordaccuracy on the 200 annotated pages turns out to be quitelow (64%), which suggests that there is scope to improvethe word accuracy using our batch correction techniques.Our main objective is to demonstrate that as we increasethe collection size, our automated batch correction methodbecomes better at picking the right candidate. For this, weperform clustering on data of different sizes, where wekeep on increasing the number of words for each subset.

We observe from Figure 7 that the word accuracy for thedataset improves as the size of collection increases. Thisimplies that for the larger unannotated data, the proposedbatch correction method will lead to a better improvementin word accuracy and thus reduction in overall correctioncost.

Performance of the traditional methods for error correc-tion does not change with the size of the collection. Ourmethod scales well to large collections and yields superiorperformance, making it an ideal candidate for large scaleefforts like digital libraries.

VI. CONCLUSION

In this work we propose a cost efficient batch correctionscheme involving a human editor. We also propose a novelclustering schema to improve the homogeneity of clusterswhich leads to significant reduction in correction cost. Wecompared our method with various baseline approaches. Wealso demonstrate the scalability of our batch correction ona large digitization effort. As part of our future work, wewould like to incorporate active learning techniques in orderto filter out only those batches that need human inspection,while rest of the batches will be corrected automatically.

REFERENCES[1] Project Gutenberg. www.gutenberg.org.[2] Google Books. https://books.google.co.in/.[3] Pingping Xiu and Henry S Baird. Whole-book recognition using

mutual-entropy-driven model adaptation. In DRR, 2008.[4] Karen Kukich. Techniques for automatically correcting words in text.

ACM-CSUR, 1992.[5] Youssef Bassil and Mohammad Alwani. OCR Post-Processing Er-

ror Correction Algorithm using Google Online Spelling Suggestion.CoRR, 2012.

[6] Youssef Bassil and Mohammad Alwani. OCR context-sensitive errorcorrection based on Google Web 1T 5-gram data set. AJSR, 2012.

[7] Rohit Saluja, Devaraj Adiga, Parag Chaudhuri, Ganesh Ramakrishnan,and Mark Carman. Error Detection and Corrections in Indic OCRUsing LSTMs. In ICDAR, 2017.

[8] Ray Smith. Limits on the application of frequency-based languagemodels to ocr. In ICDAR, 2011.

[9] Gregory B Newby and Charles Franks. Distributed proofreading. InJCDL, 2003.

[10] Luis Von Ahn, Benjamin Maurer, Colin McMillen, David Abraham,and Manuel Blum. recaptcha: Human-based character recognition viaweb security measures. Science, 321, 2008.

[11] Ahmad Abdulkader and Mathew R Casey. Low cost correction ofOCR errors using learning in a multi-engine environment. In ICDAR,2009.

[12] Henry S Baird, Venugopal Govindaraju, and Daniel P Lopresti.Document analysis systems for digital libraries: Challenges andopportunities. In DAS, 2004.

[13] Kazem Taghva, Julie Borsack, and Allen Condit. Evaluation of model-based retrieval effectiveness with OCR text. ACM-TOIS, 1996.

[14] Praveen Krishnan and CV Jawahar. HWNet v2: An Efficient WordImage Representation for Handwritten Documents. arXiv preprintarXiv:1802.06194, 2018.

[15] Richard O Duda, Peter E Hart, and David G Stork. Pattern classifi-cation and scene analysis. Wiley New York, 1973.

[16] Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors:towards removing the curse of dimensionality. In ACM-STOC, 1998.

[17] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainableneural network for image-based sequence recognition and its appli-cation to scene text recognition. In PAMI, 2017.