Silver Linings Playbook: Intuit's MT Journey

Fri Oct 11 9am

Render Chiu, IntuitGroup Manager, Global Content & Localization

Tuyen Ho, WelocalizeSenior Director

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serve informational and educational purposes only.

MT Journey Outcome?

MT in 3 Months?

Silver Linings Playbook is a 2012 American romantic comedy-drama film written and directed by David O. Russell, adapted from the novel The Silver Linings Playbook by Matthew Quick. Reference is for informational purposes only.

• $4.15 billion rev in 2012 • Flagship products: QuickBooks,

TurboTax and Quicken• New: Mint.com, Intuit Money Manager• Markets: North America, Europe,

Singapore, Australia, India

Globalization Business Driver: Opportunity to Serve a Global Ecosystem

Business & Technical Landscape• Focus: QuickBooks Online Software• Localization Readiness

• Limited i18n of the codebase• In-house team for French Canadian

only• Architecture

• WorldServer SaaS• Mix of various DBs, authoring tools

and CMS• GCL Platform

• Go-to-Market Goals• Aggressive goal to SimShip 10 to 20

languages as fast as possible

Team & Products - Today

Tax2 FTE

Mobile1 FTE

QBO3 FTE

Payroll 2 FTE

SCM process – QBO/QBO-P Build Process -- 1 FTE

Simplified English – 1 FTE Platform & Tools – 2 FTE

Internal Translation (CA French) – 4 FTE

External Translation – 2 FTE

9 Writers, 4 Translators, 2 ENG – 13 Products

QUALITY ?

Why MT?

SPEED

SCALE

COST

✔✔✔

(sure hope so)

Business Case for MT+Post-EditingBenefits

• Efficiencies– 5-100% productivity increase

• Target cost savings– 30% lower translation rates

• Faster time to market– Needed to launch in less than

4 months

• Quality– No compromising on UI

content

Considerations• Is our UI and UA content

suitable• How much do we need to

invest in engine training• What efficiency is needed to

justify the investment• What about language pairs &

productivity, e.g. FIGS higher than CJK?

• What tradeoffs do we need to be prepared to make in terms of quality vs cost

Challenges (or Reality Check)How do you go global ASAP when you start from ground zero?

Requirement StatusBilingual translations None, except for FR-CAIn-house MT expertise NoneMT engine/technology NoneTMS + MT connector NoneStructured Content One Major Plus We Had

Going for Us: STE

11

Why Simplified Technical English (STE)?

• It’s the international standard• Widespread adoption; started in the aerospace

industry, but not limited to that any more• Actively maintained and enhanced• Several checker tools that support it• More precision, less ambiguity• Easier to understand (esp. by non-native English

speakers• Easier and cheaper to translate due to clear,

unambiguous glossary and sentence structure

What Were Our Options Then?

Extreme Options We Chose Collaboration

• Lower cost by spreading the risk• Speed w/ immediate expertise• Scalability via deep supply chain

Comprehensive MT Approach Drives Quality Output

Welocalize has a multi-tiered approach to machine translation (MT) implementation:1. Evaluate content for MT readiness

– source content audit – pre-translation editing– style and glossary verification

2. Assist in selection and integration of one or multiple MT engines into the localization technology ecosystem

3. Perform MT post-editing services– evaluation of MT output quality via workbench– human assessment and automated scoring– engine training feedback / engine improvement

4. Support transition from SaaS/hosted “black box” model to hosted glass box or in-house model

Req. gathering

Solution Architecture

Engine Training

Feedback Loop(s) PE Metrics “Go Live”

Intuit – Welocalize – MT Engine Coordination:

1) Client formulates the program requirements

2) MT provider, LSP and client define the solution architecture

3) MT or LSP provider trains the engine• linguistic training• metadata analysis• workflow architecture• feedback loops with automated scores• human PE measurement and assessment

4) LSP calculates PE metrics

5) MT-PE projects go “live”

Ensuring Quality with MT+PE

Engine Strategy: SaaS, Trained Use Microsoft Translation Hub engine to achieve immediate cost savings and productivity gains • Automated engine training process, with minimal human involvement • No additional investment required

Pros• Cost-effective• Rapid deployment

Cons• Less control over engine training and tuning• Potentially lower productivity gains due to engine customization limitations

Segmentation & TM

propagation

TM

Translated files

uploaded; project

complete

Translation Project

(XLIFF file w/TM

propagated for X%

matches and higher

Target

Files

Source

Files

MT engine invoked for

non-TM segments

Translation

complete (TM + MT)

Post-editing

MT server

Linguistic settings

TMTM

1

2

3 4 55

6

778

Terminology

257

Translation Project

(XLIFF file w/TM

propagated for X%

matches and higher

Source

Files

MT engine invoked for

non-TM segments

Translation

complete (TM + MT)

Source TMS Translationn MT with Post-Editing

Engine Integration into L10N Ecosystem

• Language teams familiarized with MT environments

• Talent selection and testing is the key

• Human quality assessment is performed in a structured non-subjective environment

• Post-editing throughput figures are captured by iOmegaT and subsequently analyzed

• Translators realize the other benefits of the MT-based process: terminology consistency, predictability of errors, higher degree of control over the integrity of translation

Post-Editing Philosophy

Initial Results with 1 Engine Training

PT-BR ES-ESZH-CN DA NL ID DE IT -

10

20

30

40

50

60

70

BLEU

BingHub

PT-BR ES-ESZH-CN DA NL ID DE IT -

10

20

30

40

50

60

70

GTM

BingHub

Bootstrap ApproachFast

• Adopted SaaS MT ready-to-go engines with pre-populated financial domain- specific data

• Created minimum training data with 3K glossary entries and 4.5K TU for first training

Cheap

• Experimented with different free engines for branded and support site to gather feedback from customers, test markets, and identify quality gaps

Let’s Give it a Try• Leveraged pre-built

MT connector• Applied automatic &

human scoring to only a subset of translated data

MT Journey Recap

RFP Process2 months

May 2012

July 2012

Sep 2012

Nov 2012

Jan 2013

March 2013

Confirmed Target Languages4.5 months

Created Training Data

3 Months

Deployed MT Connector, Workflows, Engines + 1 Training

2.5 – 3 months

Requirements or Scope Change

10 Engines & Post Editors Ready for Any

Content

Lessons Learned

• Good wine comes from great grapes

• You can hire a professional tennis player to play for you

• You need a great team and a great partner

Looking Forward

• Continue investment on MT quality

• Evaluate maintenance & sustainability, e.g. re-training existing engines for improved performance

• Expand beyond 10 languages• It’s not all about text

Questions?

Contact:Tuyen Ho

www.welocalize.comtuyen.ho@welocalize.com