29
Industry Shared Metrics with the TAUS Dynamic Quality Dashboard and API www.taus.net Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 163
Industry Shared Metrics with the TAUS Dynamic Quality
Dashboard and APIwww.taus.net
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 163
What About Translation Quality
Old School: “One size fits all”Since the 1980’s
LISA QA Model, SAE J2450 prescribe today’s quality processes:
1. Static:• One quality fits all purposes, all content, all audiences
2. Subjective:• Evaluations are often subjective and anecdotal
3. Costly:• QE causes friction, delays• QE can cost up to 25% of total translation costs
4. Non-transparent:• Necessity without remedy
DynamicQualityFramework
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 164
Industry Collaborative ProgramDQF started in 2011
AdobeAppenAutodeskAVBCA TechnologiesCiscoCrestecCrosslangDellDFKIeBayEMCGoogleHewlett PackardIntelLDS ChurchLingo24
This slide may not be used or copied without permission from TAUS
Participating members
LionbridgeMedtronicMicrosoftMoraviaNikonOraclePacteraPangeanicPaypalPhilipsPTCSiemensSpil GamesSystranVMwareWelocalizeYahoo!
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 165
From DQF Tools to Quality Dashboard
DQF ToolsSince January 2014
Tools on TAUS web site:to measure:• Productivity• Adequacy• Fluency
to review and count:• Translation errors
to get:• Stats and reports
Used by 100+ members
Quality DashboardLaunched June 2015
DQF integrated in:• CAT Tools• TMS Systems
Use of DQF plug-in provides:• Enhanced statistics• Benchmarking
Open to everyone
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 166
The Power and Value of the Quality Dashboard
• DQF collects data and generates reports on the Dashboard real-time
• Translators, managers, buyers, developers get their own stats, benchmarks and analytics
• Not only track and benchmark against your own data, but also against industry averages, between translators, customers, projects, technologies
Quality Evaluation Business Intelligence
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 167
i
>
August 31, 2015: 566,987,756 words have been measured
287
0
50
100
150
200
250
300
350
Jan-00
Nu
mb
er
of
Wo
rds
per
Ho
ur
ProductivityAcross all languages, industries, technologies, processes,
content, translators
Quality Dashboard
ProductivityEfficiencyAdequacyFluency
LanguageTimeTechnologyProcessContentIndustryProjectTranslator/vendorCustomer
StatisticsLanguage
Time
Technology
Process
Content
Industry
Translator/vendor
Customer
Distribution of segments
LanguageTimeContent
Industry
Project
Quality Dashboard
Content profiling
Quality evaluation
Adequacy/Fluency
Error review
MT Ranking
Productivity measurement
DQF API
Home Services Academy Events Blog Membership My TAUS
Benchmark
More information
Review
What is the average productivity?
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 168
i
>
August 31, 2015: 566,987,756 words have been measured
Quality Dashboard
Quality Dashboard
Content profiling
Quality evaluation
Adequacy/Fluency
Error review
MT Ranking
Productivity measurement
DQF API
API Integrators
Home Services Academy Events Blog Membership My TAUS
Benchmark
More information
Review
ProductivityEfficiencyAdequacyFluency
LanguageTimeTechnologyProcessContentIndustryProjectTranslator/vendorCustomer
StatisticsLanguage
Time
Technology
Process
Content
Industry
Translator/vendor
Customer
Distribution of segments
LanguageTimeContent
Industry
Project
0
50
100
150
200
250
300
350
400
450
1 2
Chart Title
What is the average productivity of MT vs. TM?
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 169
i
>
August 31, 2015: 566,987,756 words have been measured
Quality Dashboard
Quality Dashboard
Content profiling
Quality evaluation
Adequacy/Fluency
Error review
MT Ranking
Productivity measurement
DQF API
Home Services Academy Events Blog Membership My TAUS
Benchmark
More information
Review
287
345
0
50
100
150
200
250
300
350
400
1 2
Nu
mb
er
of
Wo
rds
per
Ho
ur
My Productivity Compared to Industry Average
ProductivityEfficiencyAdequacyFluency
LanguageTimeTechnologyProcessContentIndustryProjectTranslator/vendorCustomer
StatisticsLanguage
Time
Technology
Process
Content
Industry
Translator/vendor
Customer
Distribution of segments
LanguageTimeContent
Industry
Project
What is my productivity compared to industry?
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 170
i
>
August 31, 2015: 566,987,756 words have been measured
Quality Dashboard
Quality Dashboard
Content profiling
Quality evaluation
Adequacy/Fluency
Error review
MT Ranking
Productivity measurement
DQF API
Home Services Academy Events Blog Membership My TAUS
Benchmark
More information
Review
20%
40%
20%
15%
5%
My Distribution of segments
1 2 3 4 5
ProductivityEfficiencyAdequacyFluency
LanguageTimeTechnologyProcessContentIndustryProjectTranslator/vendorCustomer
StatisticsLanguage
Time
Technology
Process
Content
Industry
Translator/vendor
Customer
Distribution of segments
LanguageTimeContent
Industry
Project
Where do my translations come from?
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 171
i
>
August 31, 2015: 566,987,756 words have been measured
Quality Dashboard
Quality Dashboard
Content profiling
Quality evaluation
Adequacy/Fluency
Error review
MT Ranking
Productivity measurement
DQF API
Home Services Academy Events Blog Membership My TAUS
Benchmark
More information
Review
0%
20%
40%
60%
80%
100%
120%
1 2
My Distribution of Segments Compared to
Industry
Series1 Series2 Series3 Series4 Series5
ProductivityEfficiencyAdequacyFluency
LanguageTimeTechnologyProcessContentIndustryProjectTranslator/vendorCustomer
StatisticsLanguage
Time
Technology
Process
Content
Industry
Translator/vendor
Customer
Distribution of segments
LanguageTimeContent
Industry
Project
Where do my translations come from vs. industry?
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 172
i
>
August 31, 2015: 566,987,756 words have been measured
Quality Dashboard
Quality Dashboard
Content profiling
Quality evaluation
Adequacy/Fluency
Error review
MT Ranking
Productivity measurement
DQF API
Home Services Academy Events Blog Membership My TAUS
Benchmark
More information
Review
0
50
100
150
200
250
300
350
400
1 2 3 4
Nu
mb
er
of
wo
rds
pe
r h
ou
r
Projects
Productivity by Language by Project
Series1
Series2
Series3
Series4
Series5
ProductivityEfficiencyAdequacyFluency
LanguageTimeTechnologyProcessContentIndustryProjectTranslator/vendorCustomer
StatisticsLanguage
Time
Technology
Process
Content
Industry
Translator/vendor
Customer
Distribution of segments
LanguageTimeContent
Industry
Project
What is my productivity by language and by project?
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 173
TAUS DQF SERVER
CAT Tool /
editor
Translator
TMS / GMS
CAT Tool
Project manager
Project settings
TAUS DQF Infrastructure
DQF Analysis
Engine
DQF Reporting
Engine
User
Management
Translator Manager Buyer
QUALITY DASHBOARD
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 174
Milliseconds per segment
Source segment
Target segment
Edited target segments
Time
Language pair
Project key
Translator key
DQF Data Instrumentation
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 175
Open API
Test Environment
https://dqf.taus.net/assets/api/v1/index.html
Open API on GitHub
http://github.com/TAUSBV/dqf-api
Specification
Test Code
Documentation
Issue Tracker
Available under the MIT Open Source License
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 176
Quality Dashboard Integrators
“Microsoft Office International team is committed to the DQF model and approach and are
actively partnering with TAUS to investigate how best to integrate TAUS Quality Dashboard
API into our translation tool set.”
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 177
The TAUS Efficiency ScoreIntroducing a new score for measuring productivity
2 Core variables:
Words per Hour - WPH Edits per Hour - EPH
Efficiency = WPH + EPH Normalized using Min-Max
Credit: Nikos Argyropoulos
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 178
Productivity
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 179
Productivity
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 180
Productivity
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 181
Edit Distance
Levenshtein distance
The Levenshtein distance calculates how many operations
are necessary to modify one sentence into another one.
The number of single character edits (insertion, deletion,
replacement) needed, is called the Levenshtein distance.
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 182
Efficiency = WPH + EPH
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 183
Min-Max Normalization
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 184
Normalized scores & Efficiency Score
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 185
Post-editor profiles
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 186
Post-editor profiles
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 187
Limitations and further work
More data for benchmarking
From relative to absolute scores
0 score theoretically possible = discouraging
Eliminating outliers
Additional variables to include
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 188
Additional variables to include
Keystrokes – number of keystrokes
Mouse clicks – number of clicks
TM fuzzy – 0-100%
MT confidence – 0-100%
Quality – Review, automatic QA or manual QE
Difficulty of Source
Experience – number of words produced
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 189
Harmonized error-typology
DQF & MQM HarmonizationCooperation with DFKI to harmonize DQF with MQMand standardize Error categories and metrics. A deliverable in the EU project Q21.
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 190
This slide may not be used or copied without permission from TAUS
THANK YOU!
Proceedings of MT Summit XV, vol. 2: MT Users' Track Miami, Oct 30 - Nov 3, 2015 | p. 191
