The Measurement of Quality in Translation

The Measurement of Quality in The Measurement of Quality in TranslationTranslation

42nd ATA Annual Conference – Los Angeles 2001

Riccardo Schiaffino – President – Aliquantum

Franco Zearo – Worldwide Director, Language Services – Lionbridge Technologies

© 2001 by J.D. Edwards and Lionbridge Technologies, Inc.

OverviewOverviewOverview

Technical translation and qualityTranslation quality initiativesQuality Control vs. Quality Assurance

Our proposal for quality assuranceChecklistsSampling techniques

ConclusionsImportance of cost/benefit factors

Translation Quality InitiativesTranslation Quality Initiatives

ISO 9002EUATC Quality StandardDIN 2345ASTM Standard for Language Translation

SAE J2450LISA QA Model

Academic translation theories and studiesPrivate sector methodologies

Product & Process AssessmentProduct & Process Assessment

Translation quality assessment must apply to both:

The translated text(the “product”)

The translation process(the “process”)

QC vs QAQC vs QA

Quality Control (QC)Quality verification over the whole text. Example: editing.

Quality Assurance (QA)Sampling techniques, control of quality over a (statistically significant) sample of the whole text.Example: quality measurement.

Why is Quality Measurement Important?Why is Quality Measurement Important?

You can’t manage what you can’t measureIt is difficult to improve something if you cannot measure it.

Such measurement should be repeatable and objective.

Different persons should arrive at similar assessment for the same piece of translation.

Our Definition of QualityOur Definition of Quality

Functional approach to quality

Different views of translation lead to:Different concepts of quality Different assessments

Quality is defined as meeting the needs and expectations of the customer or user.

Customer-driven ConsiderationsCustomer-driven Considerations

Conformance to specificationsCustomer’s vs. One’s own

Fitness for useHow well the translation performs its intended purpose

Value ( = quality & price)How well the translation performs its intended purposeat a price customers are willing to pay

SupportE.g.: Printing, testing

Psychological impressionsE.g.: In-country translators; certification

Translation Quality Factors Translation Quality Factors

Usability Accuracy

Legal Marketing

Quality Measurement: Our ProposalQuality Measurement: Our Proposal

What Can Other Disciplines Teach Us?Use checklists to collect the data

Identify types of errors, issues or problemsDetermine relative importance of issues (may be different for different languages; e.g., spelling errors in English, French or Italian)

Use sampling techniques to assess your quality levelDetermine percent thresholds for various levels of qualityDetermine whether you have achieved your target quality or not

Use of ChecklistsUse of Checklists

There are several quality assessment methodologies that rely on the use of checklists – among these the LISA methodology.

Language: Reviewer: Date: Result: Pass Comments:

Client NameProject NameProject NumberProject Manager

Critical max. error points + 1Number of words 0 Major 5 pointsMax error points allowed 0 Minor 1 point

Error Category Minor Major Critical total max. allowed0 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 0

Total 0 0More elaborate descriptions of the error criteria can be found in the LISA QA model version 1.0 Reference Manual.

Quality Assurance Form

Consistency

MistranslationAccuracyTerminologyLanguageStyleCountry

Use of ChecklistsUse of Checklists

There are several quality assessment methodologies that rely on the use of checklists – among these the LISA methodology.

We would like, however, to advocate the use not of “universal”checklists, but of checklist specifically tailored to each language.

Checklists for evaluating translation companies

Checklists and tests for evaluating translators

Checklists for evaluating translationsLimitations of universal checklists

Language specific checklists (example, different weight of spelling correctness for different languages)

Statistical MethodsStatistical Methods

Defect CountsStatistics on Effort Per Defect

Defect Density Prediction

Defect Pooling

Defect Seeding

Defect CountsDefect Counts

Useful to obtain a quantitative measurement of how much QC work to do.

Ratio of new defects to defects solved.

Statistics on Effort Per DefectIn order to estimate the scope of the defect correction work, itis necessary to have good data on the time necessary to fix the various types of defects

Defect Density PredictionDefect Density Prediction

One way to judge whether the QC work on a translation project is complete is to measure its defect density (the number of defects per page, per 1,000 words or per screen).

Defect PoolingDefect Pooling

Defect pooling is a simple defect prediction technique that separates the defects found in a translation sample into two pools.

Depending on the number of defects found in either of the two pools (but not in both) it is then possible to estimate the defects that have not been found in the sample.

This number can then be used to estimate the number of defects in the entire project.

Defect SeedingDefect Seeding

Defect seeding is a statistical technique in which a sample of a population is extracted and used to estimate the total population.

The technique works by deliberately inserting (“seeding”) defects in a complete translation that will be QCed.

The ratio of the seeded defects found compared to the total number of defects seeded provides a rough estimate of the total number of translation defects yet to be found.

A common problem with this type of technique is forgetting to remove the errors deliberately inserted.

Inspection Points Inspection Points

Key Principle: Reject “defective material” at its lowest value

Proof

Edit

Translation

SLContent Development(GIGO)

$ Va

lue

of S

ervi

ce

Stages of Production

Customer-driven Considerations Customer-driven Considerations

Price Time

Quality

Importance of Quality Importance of Quality

Quality as a Competitive Weapon

Cost Time

Quality

Good Quality Higher Profits

Good quality of translation (product) and service (process)can pay off in higher profits

Improving on quality can reduce costs andspeed up time-to-market

Cost/Benefit AnalysisCost/Benefit Analysis

Quality measurements are a tool to determine the optimal level ofquality.

They could help us identify a cut-off point.

010203040

5060708090

1 2 3 4 5

Quality Level

Tim

e or

Mon

eyInvestmentValue Added

ContactsContacts

Riccardo_Schiaffino@aliquantum.biz – Aliquantum, Inc.

Franco.Zearo@lionbridge.com – Lionbridge Technologies, Inc.

For more information about translation issues, visit

TranslationQuality.com

Biographical Notes on the AuthorsBiographical Notes on the Authors

Riccardo SchiaffinoRiccardo Schiaffino worked as translator, translation manager and special software translation project lead for a major software company, and now leads a small company he established with a few experienced colleagues. As a translation manager, Riccardo worked on the improvement of translation quality and on translation quality metrics and tools. He holds an MA degree in Translation, and has been working in translations for over 20 years, first in Italy and then in the U.S. Riccardo is ATA accredited.

Franco Pietro ZearoFranco Pietro Zearo is a project manager with Lionbridge Technologies in Boulder, Colorado. He holds a degree in translation from the Advanced School of Modern Languages for Translators and Interpreters at the University of Trieste, Italy, and earned an MBA from the University of Phoenix. Before joining Lionbridge in 1996, he worked as a freelance technical translator in Italian, English, and Russian. At Lionbridge, he has held positions in translation, localization analysis, presales, and cultural and globalization consulting. He has been responsible for translation quality on numerous projects for many Fortune 500 clients. In his previous role as senior technical translator, he helped define best practices for the translation department.