Quality and Localization Effectiveness

QUALITY AND LOCALIZATION ‘EFFECTIVENESS’

• Let’s focus on ‘Effectiveness’ – how could that be measured….

i) Correctly functional – equal to the originalii) Fully understandable – equal to the originaliii) Saleableiv) Profitable

• The story so far….. Deming’s 14 Points

• Agile v Waterfall, Project v Continuous, Cost of Ownership v Benefit, QA Data v Business Data

TAUS GLOBAL LEADERS - DUBLIN 2016


Localization Quality Planning – remember it’s DYNAMIC

1. Pre-ProductionProcesses, Standards, Qualified/Trained Resources…

2. ProductionControl, Procedure, Management, Tools..

3. Post-ProductionMeasurement, Data, Analysis, Feedback



• DQF in the overall scheme…


AND FINALLY…..

4. Localization Quality Improvement….

Why do we care about quality?

Bad quality leads to complaints.

Complaint management is a terrible job.

Quality is hard to quantify, but expertise defeats complainers.

You only have to be better than your peers.


What do you do to improve quality?

You give better reference materials.

You train your vendors.

You change your vendors when they don’t follow your requests. Are you able to get better vendors for the same amount of money? As one company, you may. As an industry, you won’t.

You train your own people. Translation is not all about outsourcing.


Vendor management requires more than pass/fail

You need actionable reports. You need to know where you need training to

your vendors. You need your vendors to train your

subvendors. You need to know where you fail in your

internal processes and get the buy-in. Internal and external training means there is

no one-size-fits-all model. Quality is compliance.


How QA models fit the purpose?

• LISA QA model was a revelation when I read it. A coursebook.

• Who should be using DQF? The beginners who need a starting point and take it from there.

• Why is DQF very current? Because localization becomes a function for business users, rather than experts.

• Benchmarking and reporting are key, just like in business.


Machine vs human translation

• Quality for an MT user is a very different story.

• Computers don’t fail to read the instructions, they fail in understanding them.

• Is it worth using the same quality model for both translation approaches?


Some Practical Thoughts

Guylaine Tritton


Planning

Glossaries

Style guides

Translator Training

Product knowledge

Client requirements

Direct contact between Client and LSP

Work with same linguists


In-Project QA

Proofreading

Automatic QA checks

Default checks

Customised checks


Scoring Machine Translation

Evaluate MT output

Reactive Rework?

End-user complaints Distributor complaints

Loss of business to competition? Savings through financial penalties on LSP?

Best practice Know that you are producing good quality Train your LSP

Because technology makes it possible


Benchmarking

Do you trust the benchmarking data?

Even if you do, are your criteria the same?

Not enough to have a Quality Program in place?

Not enough for your LSP to have a Quality Program in place?


Alpha CRC Clients vDQF-MQM Error Typology

*Similar description used

High-level error types Used by clients (out of 10)

Accuracy 5

Fluency 6*

Terminology 6

Style 6

Design 3*

Locale convention 3*

Verity 0



High-level error types Granular error types Used by clients (out of 10)

Accuracy Addition 3

Omission 3

Mistranslation 5

Over-translation 1

Under-translation 1

Untranslated 5

Improper exact TM match 1




Fluency Punctuation 5

Spelling 5

Grammar 5

Grammatical register 0

Inconsistency 0

Link/cross-reference 1

Character encoding 0




Terminology Inconsistent with termbase 5

Inconsistent use of terminology 6

Style Awkward 2

Company style 2

Inconsistent style 3

Third-party style 1

Unidiomatic 5




Design Length 0

Local formatting 1

Markup 2

Missing text 0

Truncation/ text expansion 1




Locale convention Address format 0

Date format 0

Currency format 0

Measurement format 0

Shortcut key 0

Telephone format 0

Verity Culture-specific reference 0


Interpreting data

Visual analysis different for each client

Data

Filters

Timeline

Charts

How are statistics used?


Final Thoughts

How much does QA cost?

Are a planning phase and a good LSP enough?

Are review comments not more important than scoring?

How much effort is involved in adopting TAUS DQF?


Automatic Measurements

Dragos Munteanu


Automatization helps

Planning: Evaluate vendors/translators/MT engines Determine price

In-project: QA checks Smart routing of content through workflows

Post-project: Select samples for human review

Improvement Gather data, improve metrics


Many applications, many approaches

Different applications of automatic measurements require different technologies.

The industry already has a few distinct ones:

Translation quality estimation (BLEU/METEOR/etc)

MT quality estimation (KantanAnalytics, TrustScore)

Fair pricing estimation (Fairtradetranslation.com)

More new techniques are needed!


Translation quality evaluation: a different approach

• BLEU/METEOR/TER

• Computes distance to closest human reference translation

• HTER

• Computes distance to perfect translation created on the fly


trans1

trans2

ref1

ref4

ref2 ref3

trans3

postedit1

postedit2

Poor representation of the space of good translations!

Translation quality evaluation: a different approach

• HyTER

– Efficiently encode all correct translations, and compute minimum distance


trans5

trans4

trans6

trans1

trans2

ref1

ref4

ref2 ref3

trans3

postedit1

postedit2

How to encode all translations?

Potential applications

Assess MT quality

Predict Human Translation Profficiency

Predict impact of MT on post-editing speed


Date post:	08-Jan-2017
Category:	Presentations & Public Speaking
Upload:	taus-enabling-better-translation
View:	171 times
Download:	7 times

Quality and Localization Effectiveness

Presentations & Public Speaking