WatsonThe IBM Jeopardy! Challenge

Jim De Piante, PMPExecutive Project Manager

IBM Research© 2011 IBM

Video clip 1

WatsonThe IBM Jeopardy! Challenge

Jim De Piante, PMPExecutive Project Manager

IBM Research

© 2011 IBM

© 2011 IBM 4

Watson wins

February 16, 2011

© 2011 IBM 5

At a restaurant in upstate New York

Charles Lickel

Ken Jennings

© 2011 IBM 6

Deep Blue

May 11, 1997

© 2011 IBM 7

The next Grand Challenge

• Difficult problem• Broader audience• Scientific community• Relevant to clients

© 2011 IBM 8

IBM Research pursues the idea

Paul Horn

Dave Ferrucci

© 2011 IBM 9

What they were in for

2 hours 3 seconds

Wrong 66% Right 90%

Simple questions Quirky clues

© 2011 IBM 10

Dave Ferrucci meets Harry Friedman

Harry Friedman

© 2011 IBM 11

Early sparring match

© 2011 IBM 12

Sparring matches with champions

Todd Crain

Former Jeopardy! champions

Video clip 2

© 2011 IBM 14

Watson’s record

Ken Jennings Brad Rutter

© 2011 IBM 15

Watson was a contender

Press conference after the practice match

© 2011 IBM 16

Show time

© 2011 IBM 17

All there to witness…

© 2011 IBM 18

End of round 1

© 2011 IBM 19

The result

© 2011 IBM 20

Watson

WatsonThe role of Project

Management© 2011 IBM

© 2011 IBM

• Scope• Schedule• Budget• Risk• Stakeholders• Communications• Change• Procurement• Integration

Scope© 2011 IBM

© 2011 IBM 24

Scope Win on Jeopardy!• Against the best of the best• In early 2011

Radically new technology• Architecture• Software

HW configuration Marketing / Communications Scientific research• 55 sparring matches• Research papers

Asset to commercialize

© 2011 IBM 25

Unlike any other project One of a kind• Unprecedented• No “methodology”• SW development, but not

Division of responsibilities• Principal Investigator• Project Managers

IBM infrastructure not suited Creative use of PM techniques• Flexibility• Non-rigorous

© 2011 IBM 26

Principal Investigator Deep QA software• Architecture• Algorithms • Development• Integration

Watson hardware• Configuration• Scale-out• Optimization

Front end• Game strategy• Betting• Watson’s voice• Game state

David Ferrucci, PhD

© 2011 IBM 27

Project Managers1. Project communications2. Procurement3. Stakeholder management4. Elements of the budget5. Milestone management6. Dependencies7. Risk management8. Sparring matches9. Contestant management10. Negotiations11. Change management12. Hardware build-out13. Move to Hollywood14. The set at IBM Research15. Integration of all elements

David Shepler, PMP

Jim De Piante, PMP

Schedule© 2011 IBM

© 2011 IBM 29

One unyielding deadline Early 2011 Public commitment IBM Research: Grand Challenge Hollywood: Ratings

Jeopardy! Challenge …..……|………………………|……..

© 2011 IBM 30

Shelf life of clues

Budget© 2011 IBM

© 2011 IBM 32

Budget Severely constrained• Immutable deadline• Fixed scope• Financial pressures

Managed at several levels Principal Investigator• Development resources• Department travel

Project Managers• Hardware• Other travel• Procurement

Risk© 2011 IBM

© 2011 IBM 34

Key risks Technology Reputation (brand) Schedule Perceptions• Might seem too easy• If Watson wins• If Watson loses• HAL-like associations

Failure• Show might not air• Watson could lose

Move Watson to Hollywood

Rep. Holt beats Watson?

Not!

© 2011 IBM 35

Perceptions Negative perceptions• Jeopardy! (Sony)• IBM

Risk analysis• Probability: high• Effect: high• Position: avoid

Actions• Avatar• Voice• Publicity

© 2011 IBM 36

Move to Hollywood Watson damaged in transit• Probability: low• Effect: devastating• Position: mitigate

Risk analysis• Likelihood of accident• Likelihood of damage

Actions considered• Second system• Fly/drive• Choice of supplier

Action: change approach

Stakeholders© 2011 IBM

© 2011 IBM 38

Stakeholder groups IBM Research Sony Pictures Ogilvy Talent IBM internal• Marketing• Legal• Procurement• Industrial Design• Systems Technology Group

© 2011 IBM 39

1 million miles

Hollywood

Yorktown Heights

© 2011 IBM 40

IBM Research Creative culture Desire for flexibility Not used to PM approach Worried about perceptions Protective of brand Communication style Contracts Science IBM

© 2011 IBM 41

Sony Pictures Creative culture Desire for flexibility Not used to PM approach Worried about perceptions Protective of brand Communication style Contracts Entertainment Hollywood

© 2011 IBM 42

What they had in common Creative culture Desire for flexibility Not used to PM approach Worried about perceptions Protective of brand

© 2011 IBM 43

Where they were different Motives for the project• Science and product revenue• Entertainment and ad revenue

Contracts Culture Communications• Internally• Externally

Communications© 2011 IBM

© 2011 IBM 45

Communications approach Speak several languages• IBM• IBM Research• Hollywood• Legal• Procurement• Marketing• IP Law

Use various means Update frequently

Change© 2011 IBM

© 2011 IBM 47

Approach to change Facilitate change• Know alternatives• Know costs

Isolate change• System: PI• Everything else: PMs

Two examples• Watson’s hand• Match venue

© 2011 IBM 48

The hand of Watson Plan• Watson receives clue• Watson buzzes electronically

Rationale for change• Avoid perception problems• Level playing field

Create mechanical hand• IBM Research facilities• Working prototype in 3 weeks

New risk• Introduced slight delay• Possibly lose the buzz

© 2011 IBM 49

Change of venue Plan• Watson goes to Hollywood• Wheel-of-Fortune set

Rationale for change• Avoid perception problems• On IBM Research turf

Create full Jeopardy! set• Huge undertaking• Watson stays in Yorktown Heights• Jeopardy! comes to IBM Research

Opportunities seized• Reduced risk of the move• Showcase IBM Research facility

Procurement© 2011 IBM

© 2011 IBM 51

Procurement Key PM focus Highly specialized needs Sole source Key areas• Talent• Compliance• Contestants

Outside the IBM norm• Processes• Skills

© 2011 IBM 52

Talent Challenge• Role of Alex Trebek • Short lead time

New world for IBM• Specification• Actor’s guild

Solution• Sole source• Known quantity

Todd Allen Crain

© 2011 IBM 53

Compliance and integrity Challenge• Game-show compliance• Challenge integrity

New world for IBM• Specification• Making payments

Solution• Sole source• Jeopardy’s supplier

Alex Hendrie

© 2011 IBM 54

Sparring contestants Challenge• Former Jeopardy! contestants• Manage the logistics• Tight scheduling

Solution• Considered “cattle call”• Solicit through Jeopardy!• Hire as consultants

© 2011 IBM 55

Contestant management Challenge• Find the right skills• Outsource completely

Solution• Sole source• Existing approach• Existing supplier• Explicit specification• Detailed process definition• Supplier web site• Supplier travel agency• Supplier disbursement• Extreme empowerment

Josiane Emorine and Alex Trebek

Integration© 2011 IBM

© 2011 IBM 57

Integration Crisp boundaries Communication Blended approach• Co-location• Virtual

Tight linkages• Researchers• Sony Pictures

Milestone driven• Clear goals• Clear deadlines

WatsonWhat next for ?

© 2011 IBM

© 2011 IBM 59

Does Watson think?

“Does a submarine swim?”

© 2011 IBM 60

What does Watson do?

Deep

© 2011 IBM 61

The future

© 2011 IBM 62

Nuance

ARMONK, NY and BURLINGTON, MA – 17 Feb 2011IBM and Nuance Communications today announced a

research agreement to explore, develop and commercialize the Watson computing system's advanced

analytics capabilities in the healthcare industry.

© 2011 IBM 63

WellPoint

INDIANAPOLIS and ARMONK, NY – 12 Sep 2011 WellPoint, Inc. (NYSE: WLP), and IBM (NYSE: IBM) announced an agreement today to create the first

commercial applications of the IBM Watson technology.

WatsonThe IBM Jeopardy! Challenge

Jim De Piante, PMPExecutive Project Manager

IBM Research© 2011 IBM