Date post: | 29-Nov-2014 |
Category: |
Documents |
Upload: | patrickcrompton |
View: | 532 times |
Download: | 3 times |
Extracting Value from Big Data in the Cloud -
Michael Newberry
Doggerland: Simon Fitch, Vince Gaffney and Ken ThomsonImage Source: drowned-landscapes.tumblr.comRoyal Society's Summer Science Blog (http://summer-science.tumblr.com/)
Big Data.
Big Data.
VOLUME (Size)
VARIETY (Structure)
VELOCITY (Speed)
Getting useful insightsfrom awkward data setsusing the most appropriate computing platform at each stage.
Dr Michael NewberryWindows Azure LeadMicrosoft UK
Machine Learning & Bayes theorem
𝑝 ( h𝑀𝑖𝑐 𝑎𝑒𝑙𝑏𝑢𝑦𝑖𝑛𝑔𝑠𝑜𝑐𝑘𝑠 𝑖𝑓h𝑒h𝑎𝑠 𝑗𝑢𝑠𝑡 h𝑏𝑜𝑢𝑔 𝑡 h𝑠 𝑜𝑒𝑠)𝑑𝑒𝑝𝑒𝑛𝑑𝑠𝑜𝑛
𝑝 ( h𝑀𝑖𝑐 𝑎𝑒𝑙𝑏𝑢𝑦𝑖𝑛𝑔𝑠𝑜𝑐𝑘𝑠 )𝑝 ( h𝑀𝑖𝑐 𝑎𝑒𝑙𝑏𝑢𝑦𝑖𝑛𝑔 h𝑠 𝑜𝑒𝑠 )
𝑝 ( 𝐴∨𝐵 )=𝑝 (𝐵∨𝐴 ) 𝑝 ( 𝐴 )𝑝 (𝐵 )
….Amazon (AMZN) calls this homegrown math "item-to-item collaborative filtering," and it's used this algorithm to heavily customize the browsing experience for returning customers…. Judging by Amazon's success, the recommendation system works. The company reported a 29% sales increase to $12.83 billion during its second fiscal quarter, up from $9.9 billion during the same time last year. A lot of that growth arguably has to do with the way Amazon has integrated recommendations into nearly every part of the purchasing process from product discovery to checkout.
http://tech.fortune.cnn.com/2012/07/30/amazon-5/
“In theory there is no difference between theory and practice; in practice, there is”.
Yogi Berra, cited in Nassim Taleb, Antifragile.
Big data techniques
NoSQL (ala MongoDB) Map-Reduce (e.g. Hadoop)
Embedded devices
Connected Devices
On Premise
Off Premise
Business Intelligence
Customers Employees, Partners
The Power of an Intelligent System
Modern Platform for the World’s Apps
Cloud OS
transforms the datacenterenables modern appsunlocks insights on any dataempowers people-centric IT
Cloud OS
flexible developmentunified dev-ops & managementcomplete data platformcommon identityintegrated virtualization
MICROSOFT
SERVICE PROVIDERON-PREMISES
1CONSISTENTPLATFORM
What Makes the Cloud OS Unique
RelationalNon-Relational Streaming
MANAGE ANY DATA, ANY SIZE, ANYWHERE
010101010101010101101010101010101001010101010101101010101010
Unified Monitoring, Management & Security
Data Movement
POLYBASE: COMBINING RELATIONAL AND NON-RELATIONAL DATAThe future of query processing
select... results set
Hadoop Data Warehouse
PolyBase
Single query for relational & Hadoop data
Process data in place
Future expansion to other data sources
Seamless: regular T-SQL command
19
20
Avoiding Lock-InWindows Virtual machines can move freely between all 3 clouds.
Windows Azure
Customer Data Center
Other Service ProvidersWindows
Virtual Machine
LocationOn-Premises On-Premises or
Service ProviderMicrosoft Cloud orService Provider
Rationale for Usage
Compliance
Scalability
Economies of Scale
Rapid Development
Complex, Legacy Applications
Compliance
Economics
TraditionalNON-VIRTUALIZED
AppliancePRIVATE
CloudPUBLIC
(Outside Firewall)
DATA PLATFORM DELIVERY MODELS
(Inside Firewall)
BALANCING ON PREMISE & CLOUDSnowline graph
A
Takeaways
1. “big data” can do some amazing stuff.2. Don’t think “big data” as much as “data needing non-
relational approaches”3. If your big data insights are probabilistic, which they often are,
have a plan to deal with variance. 4. Pick the most appropriate platform: Think “and” not “or”:
- Balance public cloud AND on-premise,- Combine “big data” with RDBMS.
Q+A