ABBYY FlexiCapture & Designing a High-Performance System at #ABBYYSummit16

transcript

ABBYY TechnologySummit2016

ABBYY NAHQ, 2016

Pierre van der Westhuizen

Introduction:Designing High Performance Systems

• Upscaling FlexiCapture, identify bottlenecks, and optimize system performance

• Performance Metrics and testing

• FlexiCapture Performance at a glance

– Scaling

– Up to 3 million pages per day

– High Fault Tolerance and Availability

Agenda:Designing High Performance Systems

• Introduction• *Architecture of FlexiCapture• Component Interaction Walkthrough• Defining Performance Metrics• Scaling of Systems (Demo, Medium, Large)• *Optimizing: Processing Stations, Scanning Stations and Workflow• Optimal Values of and Limitations on System Performance• System Monitoring and Bottleneck Detection• Performance testing• *Improving your current system

Architecture

• Application Level

• Application Server

• Licensing Server

• Processing Level

• Processing Server

• Data Storage

• Database

• File Storage

Architecture – Server Side

• User Stations

• Scanning

• Verification

• Processing Stations

• Administration/Monitoring Web Console

• Project Setup Station

Architecture – Client Side

Component Interaction

Performance Metrics

Defining Performance Metrics

We measure performance in volumes processed per period of time.

Define target performance using performance metrics:

• The required processing time

• Processing volumes

Parameters that shape workload

• Average batch size in pages

• Image color mode: color, grayscale, black-and-white

• Pages per day (i.e. 24 hours), average/peak

• Pages per hour, average/peak

• Average document size in pages

• Number of scanning operators

• Number of verification operators

• Document storage time

SCALING

Scaling

Demo System

Medium System

Medium System:Application Server

1. Fast network2. Fast connection to FileStorage and Database3. Fast CPU4. RAM

Medium System:Processing, Licensing Servers

For redundancy see:

FlexiCapture System Administrator Guide

Medium System:Database Server

• More RAM• Fast HDD• Avoid Mirroring• Separate Data and Logs• Index Updates

Medium System:File Storage

Read-write and capacity requirements depend on:

• Average and Peak processed per daySpeed Required for 10,000pages/hr2.8pages/sec = 2.8*3MB/sec = 8.4MB/sec

• Amount of time that documents are stored:e.g. 16 x 100,000 grayscale images x 3 MB (average file size for grayscale image) = 4.8 TB of data

NOTE: We strongly recommend something like RAID 10

Large System

Optimizing Processing Stations

Processing Stations

• Tune each station

• Add more stations

Processing Station:Hardware

• 1 Process per core• 16 cores max• 1GB RAM per core

Processing speed greatly depends on the CPU speed and the Hard Disk read-write speed

Processing Station: TEMP Folder

Scenario: 100 page batches on 8-core Station

• 100 pages x 3 MB = 300MB

• 8 Cores means 8 simultaneous executive processes

• TEMP Folder Space required is 300 x 8 = 2.4GB

• Allocate 2GB per Core + 2.4GB for TEMP = 18.4GB RAM

Calculate Number of Processing Stations

Estimate the required number of processing cores

Measure how long it takes to process one batch for one core

8-core Processing Station

Process:1. Create 24 copies of a typical batch2. Put all batches in the FlexiCapture hotfolder3. Start the timer at the first import task created 4. Stop timer after the last result has been exported to the backend

15 minutes elapsed

Each core has processed 3 batchesTime to process 1 batch is about 5 minutesIf batch has 69 pages => takes 4.35 seconds to process 1 page.

Estimate the required number of processing cores Cont’d

Estimate desired number of cores

Assume you need to process P pages in T time. We already know from the above that 1 core needs t time to process 1 page. Hence, you need N = (P x t ) / T cores.

Example. 200,000 pages in 8 hours = 28,800 secondsWe know 1 core takes 4.35 seconds to process 1 page200,000 x 4.35/28,800 = 31 cores=>4 Processing Stations with 8 cores (32 cores in total) will be sufficient processing.

Processing Cores – Limiting Factors

• The total load on the infrastructure that may result in bottlenecks– Server Hardware

– Network

– Shared Resources (Database, External Services)

• The number of processing cores that can be served by the Processing Server– Max 120 cores

Monitor Free Processing Cores on Processing Server

Optimizing: Scanning Stations

Scanning Stations

• Performance Limits– Scanner Speed

– Data Transfer Bandwidth

• Separate Network Interface for Scanning

• Setup scan settings– Color Mode

– Remove Blanks

• Schedule Image Uploads

Optimizing: Workflow

Workflow

• Avoid too many stages

• The slowest stage limits the performance

• Do not produce tasks that are too small when parallelizing processing at a stage

Optimal Values and Limits

Optimal Values of and Limitations on the System Performance 1

Factor Optimal values & limitations

System performance inpages per 24 hours:

Able to process:

up to 20,000 black-and-white or up to 1000 color pages per 24 hours

Medium up to 1 mln black-and-white or upto 300,000 color pages per 24 hours,using a farm of regular computers

Large up to 3 mln black-and-white or upto 1 mln color pages per 24 hours

Number of scanning operators FlexiCapture is able to host 1000scanning operators.

Number of verification operators FlexiCapture is able to host 300verification operators.

Number of processing Stations We used up to 120 cores in total for allProcessing Stations.

Number of cores per Processing Station regular disk drive: up to 8 cores.fast disk drive: up to 16 cores.RAM drive: up to 32 cores.

Number of pages in a Batch Optimal value is from 10 to 1000 pagesin a batch

Number of pages in a Document Optimal value is up to 100 pages in adocument

Number of pages, documents, and batchesin the system

This highly depends on hardware used.For a Large configuration, up to 100,000batches, or 1 mln documents, or 10 mlnpages is normal.

Data storage time Typically, pages, document, batches and event log records are stored in theSystem for up to 2 weeks.

Statistics for reporting can be stored foryears with no impact on performance

Performance Testing

• Single Entry Point Project– Import from Scanning Station

– Pre-processing

– Recognition

– Export

– Processed

System Monitoring and Bottleneck Detection

• Document processing monitoring via the Administration and Monitoring Console

• Hardware monitoring for each FlexiCapture server component using various Windows Performance Monitor counters. – Memory– CPU– Hard Disk– Network– IIS– SQL Server

Setting up Performance Counters

• Monitor FlexiCapture state and search for bottlenecks– Performance Monitor utility

• Recorded by Processing Server

• mmc /32 perfmon.msc

• Enable Counters

Improving Your Current System

Improving your current System

• Separate your Servers– App Server

– Processing Server/License Server

– Processing Station

– Database Server

Improving your current System Cont’d

• Database– More RAM is better – At least 4GB

– Fast Drives

– Data and Logs on Separate Drives

– Autogrowth – 100MB Increments

– Simple Recovery Model

– Maintenance Plans• Backups – Truncate Transaction Log

• Indexes

• Application Server– IIS Logs

– Caching

– Fast Hard Drive

– App Server Recycling Pool

– Number of Threads

– 2 NICs 1GB/s

Consider Load Balancing

• File Storage– Disable Search indexing and anti-virus scanning of FileStorage

– Do not store images in SQL database

– 1GB/s access

• Batches– Purging (2 weeks or less)

– Limit Big batches (Less than 100 pages per batch)

• Input and Output– Put input directory on Server

– Separate location for Export and Hotfolders

• Networking– Network Speed

– Switching

– VLANS

– Network Interfaces

Final Summary

• Architecture– Medium System

• How to Optimize Processing Stations– RAM

• Improving your current system– Separate your Servers!

Questions