Date post: | 08-Sep-2014 |
Category: |
Technology |
Upload: | amazon-web-services |
View: | 14 times |
Download: | 1 times |
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Mick Etoh, Ph.D. November 13, 2013
DOCOMO CLOUD
1
Friday, November 15, 13
Who am I?
2
Friday, November 15, 13
Who am I?• Mick Etoh, SVP, NTT DOCOMO
2
Friday, November 15, 13
Who am I?• Mick Etoh, SVP, NTT DOCOMO• Role: R&D Strategy
2
Friday, November 15, 13
Who am I?• Mick Etoh, SVP, NTT DOCOMO• Role: R&D Strategy • Expertise:
2
Friday, November 15, 13
Who am I?• Mick Etoh, SVP, NTT DOCOMO• Role: R&D Strategy • Expertise:
Pattern Recognition & Machine Learning, Signal Processing & Multimedia, Mobile Networks
2
Friday, November 15, 13
Who am I?• Mick Etoh, SVP, NTT DOCOMO• Role: R&D Strategy • Expertise:
Pattern Recognition & Machine Learning, Signal Processing & Multimedia, Mobile Networks
Fun:
2
Friday, November 15, 13
Who am I?• Mick Etoh, SVP, NTT DOCOMO• Role: R&D Strategy • Expertise:
Pattern Recognition & Machine Learning, Signal Processing & Multimedia, Mobile Networks
Fun:Road Bike, Fly Fishing, and Wine Lover
2
Friday, November 15, 13
Who is DOCOMO?
3
Friday, November 15, 13
Who is DOCOMO?
• $61 million subscribers in Japan
3
Friday, November 15, 13
Who is DOCOMO?
• $61 million subscribers in Japan• $45 billion revenue (2013)
3
Friday, November 15, 13
Who is DOCOMO?
• $61 million subscribers in Japan• $45 billion revenue (2013)
Next to China Mobile, Verizon, and AT&T
3
Friday, November 15, 13
Who is DOCOMO?
• $61 million subscribers in Japan• $45 billion revenue (2013)
Next to China Mobile, Verizon, and AT&T
• “i-mode” innovator (1999)
3
Friday, November 15, 13
Revenue Structure (FY2012)
Device
ServiceCloud
PacketNetwork
VoiceNetwork
Feature Phone EraVoice Stage Data Stage
4
Friday, November 15, 13
Revenue Structure (FY2012)
Device
ServiceCloud
PacketNetwork
VoiceNetwork
Smart Phone
Voice: $13 Billion
Data: $19 Billion
Service $5.5 Billion
Feature Phone EraVoice Stage Data Stage
4
Friday, November 15, 13
Wireless Carrier + Service Provider
5
Friday, November 15, 13
DOCOMO CLOUD Services
6
Friday, November 15, 13
DOCOMO CLOUD Services
docomo Phonebook docomo Mail Photo Collection databox dmarket
Schedule and Memo ToruCa Hanashite Hon'yaku Utsushite Hon'yaku dmenu
Mail-Honyaku-Concier Shabette-Concier docomo DriveNet i-concier Sugotoku Containt
6
Friday, November 15, 13
Service Delivery Platform RENEWED
7
Friday, November 15, 13
Service Delivery Platform RENEWED
DOCOMO CLOUD7
Friday, November 15, 13
Enterprises using Public Cloud in Japan
8
Friday, November 15, 13
Enterprises using Public Cloud in Japan
2.5% (2011)Yano Research
8
Friday, November 15, 13
Enterprises using Public Cloud in Japan
7.3% (2012)Yano Research
8
Friday, November 15, 13
DOCOMO as a Big AWS User
9
Friday, November 15, 13
DOCOMO as a Big AWS UserFour-DigitEC2 Instances
9
Friday, November 15, 13
A Telco Story About Launching Voice-Command Personal Agent Service with AWS Cloud
10
Friday, November 15, 13
Siri
11
Friday, November 15, 13
“Shabette-Concier” Voice agent service
12
Friday, November 15, 13
“Shabette-Concier” Voice agent service
• Launched Mar. 1, 2012• Over 40 services in it• Including chatting• 10 million users
Shabette
Voice
=
Concier
Concierge
=How may I help you?
12
Friday, November 15, 13
13
Friday, November 15, 13
13
Friday, November 15, 13
14
Friday, November 15, 13
14
Friday, November 15, 13
15
Friday, November 15, 13
15
Friday, November 15, 13
An Entrance to Services
16
Friday, November 15, 13
Train transfer / Last train alarm / Weather alarm / Traffic information / Receipt search / Map search / Weather search / Image search / QA search / Movie search / Music search / News search / Twitter search / Game search / App search / Book/Comic search / Area guide / Browser search / Artist search / TV program search / Dictionary / Horoscope / Result search for soccer game / Result search for baseball game / Result search for golf tournament / Shopping / Calculate calorie of food / Sightseeing search / Call taxi / Knowledge of life / Q&A for DOCOMO / Phone call / Launch camera / Timer / Alarm / Scedular / Memo / Music player / Email / i-Concirge / i-Channel / DOCOMO Map / Disaster kit / Ir communication / Post Twitter / Photo collection / Launch Galary / Translation / Cheat sheet / User support / Flight mode / Day and time / Terminal information / Display setting / Keyboard and input / Storage setting / Language setting / Location service setting / Application listup / Security setting / Sound setting / User dictionary / Wi-Fi setting / Wall paper setting / Battery setting / Communication service setting
An Entrance to Services
37search types
22applications
20functions
16
Friday, November 15, 13
Monetization
Just say, wanna buy!“d-shopping” : real marketplace
17
Friday, November 15, 13
Monetization
Just say, wanna buy!“d-shopping” : real marketplace
I want to buywater!
How may I help you?
OK !
17
Friday, November 15, 13
18
Friday, November 15, 13
“Status Quo”Shrine
18
Friday, November 15, 13
“Status Quo”Shrine
“99.9999%”Temple
18
Friday, November 15, 13
“Status Quo”Shrine
“99.9999%”Temple
Office of Inspirations
18
Friday, November 15, 13
“Status Quo”Shrine
“99.9999%”Temple
Office of Inspirations
WannabeSkunkworks
18
Friday, November 15, 13
19
Friday, November 15, 13
In Status Quo Shrine, so many superstitions....
19
Friday, November 15, 13
Is this ‘Cloud’?
In Status Quo Shrine, so many superstitions....
19
Friday, November 15, 13
In 99.9999% Temple,
20
Friday, November 15, 13
In 99.9999% Temple,Belief: High Availability Forever
20
Friday, November 15, 13
In 99.9999% Temple,
Downtime per Year 31.5 seconds
Belief: High Availability Forever
20
Friday, November 15, 13
Three Major Concerns to Public Cloud
Security
Performance
Switching Cost
21
Friday, November 15, 13
At Office of Inspirations
22
Friday, November 15, 13
At Office of Inspirations
Make a Magic Lamp right now
with Genie (2009)
22
Friday, November 15, 13
In Wannabe Skunkworks (2010)
23
Friday, November 15, 13
In Wannabe Skunkworks (2010)
Maybe, it’s high time to make
a voice agent.But, as a stealth product.
23
Friday, November 15, 13
In Wannabe Skunkworks (2010)
Maybe, it’s high time to make
a voice agent.But, as a stealth product.
Yep, we have lot of expertise & data for speech recognition.
23
Friday, November 15, 13
In Wannabe Skunkworks (2010)
Maybe, it’s high time to make
a voice agent.But, as a stealth product.
Yep, we have lot of expertise & data for speech recognition.
Just out of curiosity,
I’d like to use Public Cloud.
23
Friday, November 15, 13
In Wannabe Skunkworks (2010)
Maybe, it’s high time to make
a voice agent.But, as a stealth product.
Yep, we have lot of expertise & data for speech recognition.
Just out of curiosity,
I’d like to use Public Cloud.
23
Friday, November 15, 13
Distributed Speech Recognition (2006)
First Implementation of ETSI AURORA Project (2000-2003)16 Khz Sampling MFCC feature vectors --> 5.6 Kbps
24
Friday, November 15, 13
Distributed Speech Recognition (2006)
First Implementation of ETSI AURORA Project (2000-2003)16 Khz Sampling MFCC feature vectors --> 5.6 Kbps
Mobile Terminal
FeatureExtraction
Result
Encode
Server
Decode Rec.Engine
Dictionary
24
Friday, November 15, 13
Data Mining Team(2009)
In Wannabe Skunkworks
25
Friday, November 15, 13
Data Mining Team(2009)
Web 2.0 Believers
In Wannabe Skunkworks
25
Friday, November 15, 13
The paradigm shift from ‘Search’
26
Friday, November 15, 13
The paradigm shift from ‘Search’
Microsoft Bing
Google Voice Search
26
Friday, November 15, 13
A Small Team in Wannabe Skunkworks (2010-2012)
☹
AgileDevelopment
Core engineer
Core engineer Developmentpromoter
Developmentpromoter
Product owner
27
Friday, November 15, 13
A Small Team in Wannabe Skunkworks (2010-2012)
☹
AgileDevelopment
Core engineer
Core engineer Developmentpromoter
Developmentpromoter
Product owner
27
Friday, November 15, 13
Basic Architecture 2010
28
Friday, November 15, 13
Basic Architecture 2010(inspired by Microsoft Bing, and Google Voice Search)
28
Friday, November 15, 13
Basic Architecture 2010
Logging
VoiceRecognition
TaskRecognition
Logging
Voicetext text contents
ServiceProviders’ DB
contents
text
Text to speech
(inspired by Microsoft Bing, and Google Voice Search)
28
Friday, November 15, 13
5/2011Stealth Product
Data Center
(Hybrid with other Cloud)
History of Voice Agent
29
Friday, November 15, 13
5/2011Stealth Product
Data Center
(Hybrid with other Cloud)
History of Voice Agent3/2012
Mar.~
Version 1
PublicCloud 29
Friday, November 15, 13
5/2011Stealth Product
Data Center
(Hybrid with other Cloud)
History of Voice Agent
June
AWSNC-reg
3/2012
Mar.~
Version 1
PublicCloud 29
Friday, November 15, 13
5/2011Stealth Product
Data Center
(Hybrid with other Cloud)
History of Voice Agent
June
AWSNC-reg
Sept.
11/2012
AWSTokyo-reg.
Version 2
3/2012
Mar.~
Version 1
PublicCloud 29
Friday, November 15, 13
Speech
Speech Data
Language Data
Machine Learning Machine Learning
AcousticModel
Lexicon LanguageModel
Speech Recognition I want...
Text
Voice Recognition
30
Friday, November 15, 13
BELIEF: More data usually beats better algorithms.
Speech
Speech Data
Language Data
Machine Learning Machine Learning
AcousticModel
Lexicon LanguageModel
Speech Recognition I want...
Text
Voice Recognition
30
Friday, November 15, 13
Task Recognition and Service Flow
Find a good Italian restaurant in Palo Alto
Abstractor
Featureextractor
MC-SVMClassifierTokenizer
Query ext.
I want...
Text
Contents
Restaurant
Task CorpusLexiconDictionary
Abstraction Dictionary
31
Friday, November 15, 13
Task Recognition and Service Flow
PRINCIPLE: Machine learning enhances the service quality. A better quality service acquires more data.
ServiceLauncher
Search Engine A
Search Engine BSearch Engine C
Find a good Italian restaurant in Palo Alto
Abstractor
Featureextractor
MC-SVMClassifierTokenizer
Query ext.
I want...
Text
Contents
Restaurant
Task CorpusLexiconDictionary
Abstraction Dictionary
31
Friday, November 15, 13
Version 1 Implementation (Mar. 2012)
☹
Started with two local cloud providers in Japan, and soon faced two difficulties:
32
Friday, November 15, 13
Version 1 Implementation (Mar. 2012)
☹
A
LB
B
Started with two local cloud providers in Japan, and soon faced two difficulties:
32
Friday, November 15, 13
Version 1 Implementation (Mar. 2012)
☹
A
LB
B
Started with two local cloud providers in Japan, and soon faced two difficulties:
• Scalability in server count
32
Friday, November 15, 13
Version 1 Implementation (Mar. 2012)
☹
A
LB
B
Started with two local cloud providers in Japan, and soon faced two difficulties:
• Scalability in server count• Inflexible monthly
payment32
Friday, November 15, 13
Scrambling•Mar 2012 0.5M accesses/day
•Apr 2012 1.5M accesses/day
•Jul 2012 2.5M accesses/day
33
Friday, November 15, 13
Scrambling•Mar 2012 0.5M accesses/day
•Apr 2012 1.5M accesses/day
•Jul 2012 2.5M accesses/day
33
Friday, November 15, 13
Moved to AWS (June, 2012)
34
Friday, November 15, 13
Moved to AWS (June, 2012)
Scale!
34
Friday, November 15, 13
Technology Platform Leader
Innovative Ecosystem
Global User Footprint
AWS Partners
Moved to AWS (June, 2012)
Scale!
34
Friday, November 15, 13
System Architecture (June 2012)
Availability Zone #1
SmartPhone
ManagementServer
Log Server for VR Availability Zone #2
Voice Recognizer(VR) Task Recognizer(TR) Log management system
Same as AZ #1
TR Servers
ELB (across multiple zones) Tokenizer Access Log Servers
Availability Zone #3
VPC
VR Servers
LBELB(across multiple zones)
ELB(across multiple zones)
35
Friday, November 15, 13
Killer Design Pattern: Multi-Data Center
Availability Zone #1 Availability Zone #2
LBLB
VR servers
Route 53
m2.4xlarge x ~300 !!
HAProxy
Voice
VPC
VR servers
Voice Recognition Part
36
Friday, November 15, 13
DOCOMO Skunkworks: Cloud Natives• For another service,
adopted 21 out of 48 design patterns.
• Among them, typically used ‘Queuing Chain’ and ‘Scale Out’ Pa4erns.
37
Friday, November 15, 13
Other Crucial Design Patterns
38
Friday, November 15, 13
Other Crucial Design Patterns
•Scheduled Scale Out
38
Friday, November 15, 13
Other Crucial Design Patterns
•Scheduled Scale Out
38
Friday, November 15, 13
Other Crucial Design Patterns
•Scheduled Scale Out
•Clone Server
38
Friday, November 15, 13
Traffic Pattern
0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00
Normal traffic
39
Friday, November 15, 13
Traffic Pattern
0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00
Normal traffic
39
Friday, November 15, 13
Bursty traffic for which Auto-Scale failsTV commercial causes a spike traffic.
40
Friday, November 15, 13
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Bursty traffic for which Auto-Scale failsTV commercial causes a spike traffic.
30x traffic
# of servers
40
Friday, November 15, 13
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Bursty traffic for which Auto-Scale failsTV commercial causes a spike traffic.
30x traffic
# of servers
20:38 20:40 20:43 20:45
10min
40
Friday, November 15, 13
Unexpected TrafficScheduled
–TV Commercial–Introduction by TV program
fortunately pre-scheduled!
Suddenly happened–Earthquake–New Feature Release
burst
burst, but gradually tapering
41
Friday, November 15, 13
Prepare for Launching Servers
Refresh AMI every day–VR model has been changed everyday
Launch AMI for bursty traffic
–10 min - 30 min to launch•Loading VR model is taking time....
42
Friday, November 15, 13
More intelligence
24hours
43
Friday, November 15, 13
More intelligence
Optimized server deployment using Elastic Map Reduce (EMR) for pattern analysis
24hours
43
Friday, November 15, 13
Moved twice • Apr 2012–Start to plan for moving-out to AWS
• Jun 2012–Moved to AWS Northern California
• Tokyo did not have enough instances at that time
• Deploy mixture instance types (c1.xlarge and m2.4xlarge)
• Sep 2012–Moved to AWS Tokyo
• Unified to m2.4xlarge–Improved latency
44
Friday, November 15, 13
“99.9999%”Temple
“Status Quo”Shrine
Office of Inspirations
DOCOMOSkunkworks
October, 2012
(not ‘Wannabe’ anymore)
45
Friday, November 15, 13
Kid, your game is over.
“99.9999%”Temple
“Status Quo”Shrine
Office of Inspirations
DOCOMOSkunkworks
October, 2012
(not ‘Wannabe’ anymore)
45
Friday, November 15, 13
Kid, your game is over.
Use our stable system.
“99.9999%”Temple
“Status Quo”Shrine
Office of Inspirations
DOCOMOSkunkworks
October, 2012
(not ‘Wannabe’ anymore)
45
Friday, November 15, 13
Kid, your game is over.
Use our stable system.
Move onquickly! “99.9999%”
Temple
“Status Quo”Shrine
Office of Inspirations
DOCOMOSkunkworks
October, 2012
(not ‘Wannabe’ anymore)
45
Friday, November 15, 13
Kid, your game is over.
Use our stable system.
Move onquickly!
Catch meif you can SCALE!
“99.9999%”Temple
“Status Quo”Shrine
Office of Inspirations
DOCOMOSkunkworks
October, 2012
(not ‘Wannabe’ anymore)
45
Friday, November 15, 13
In DOCOMO Skunkworks (2013)
46
Friday, November 15, 13
In DOCOMO Skunkworks (2013)
They are SLOW and pricy. Keep Restless
improvement and our lean-startup culture.
46
Friday, November 15, 13
In DOCOMO Skunkworks (2013)
They are SLOW and pricy. Keep Restless
improvement and our lean-startup culture.
Yep, let’s continue our system
improvement on the fly.46
Friday, November 15, 13
In DOCOMO Skunkworks (2013)
They are SLOW and pricy. Keep Restless
improvement and our lean-startup culture.
Yep, let’s continue our system
improvement on the fly.
Hey, leader,how about multi-lingual
translation services?
46
Friday, November 15, 13
In DOCOMO Skunkworks (2013)
They are SLOW and pricy. Keep Restless
improvement and our lean-startup culture.
Yep, let’s continue our system
improvement on the fly.
Hey, leader,how about multi-lingual
translation services?
46
Friday, November 15, 13
Now, we have …
• 10 million terminals pre-installed• More than 4M distinct user accesses• 1M accesses / day
17-Nov-16 27-Nov-16 7-Dec-16 17-Dec-16 27-Dec-16 6-Jan-17 16-Jan-17 26-Jan-17 5-Feb-17 15-Feb-17 25-Feb-17 7-Mar-17 17-Mar-17 27-Mar-17 6-Apr-17 16-Apr-17 26-Apr-17 6-May-17 16-May-17 26-May-17 5-Jun-17 15-Jun-17 25-Jun-17 5-Jul-17 15-Jul-17 25-Jul-17 4-Aug-17 14-Aug-17 24-Aug-1717-Nov-12Number of Speech
47
Friday, November 15, 13
Other Tips•Instance Management✓Simple DB as a key-value
store
•Log management✓Enormous logs from servers✓Direct write to S3
•Multi-AZ✓HA and act-act configuration
48
Friday, November 15, 13
Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’
49
Friday, November 15, 13
Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’
in changing our development style to
49
Friday, November 15, 13
Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’
in changing our development style to“Deploy first, think later,”
49
Friday, November 15, 13
Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’
in changing our development style to“Deploy first, think later,”
49
Friday, November 15, 13
Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’
in changing our development style to“Deploy first, think later,”
in bringing us a high performance culture, and
49
Friday, November 15, 13
Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’
in changing our development style to“Deploy first, think later,”
in bringing us a high performance culture, and
49
Friday, November 15, 13
Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’
in changing our development style to“Deploy first, think later,”
in bringing us a high performance culture, and
in becoming “Cloud Natives.”
49
Friday, November 15, 13
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
MBL202 Thank You
50
Friday, November 15, 13