Data Science on Cloud Foundry Ian Huston @ianhuston
Alexander Kagoshima @akagoshima
Who are we?
• Data Scientists at Pivotal Labs • Using Cloud Foundry since 2013 • Working with enterprises to get value out
of their data
Image by Drew Conway: http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram
Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.
- Josh Wills
Typical Projects
Risk Analysis
Predictive Maintenance
Understanding Your Customer
Data Services Easy control of incoming data
Data Services Bind and scale system services
– Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME
Add User Provided Services – Standalone Hadoop or Apache Spark cluster,
Big Data System $ cf cups SERVICE_INSTANCE -‐p "host, port, username, password"
Data Service
App App App
App App App
Deploy a Model Prediction API Control distributed computation
h"ps://github.com/ihuston/python-‐conda-‐buildpack Install PyData packages with binary builds using conda
h"ps://github.com/alexkago/cf-‐buildpack-‐r R interpreter and package setup, ready for RShiny
Siloed Data
Siloed Systems
Distributed Big Data Platform
HOW TO DEPLOY
MODELS? Data Extract
?
(Model development
happens here!)
(Business needs model predictions
here!)
App
App
App
App
App
Big Data Platform Big Data Storage
R E S T A P I
Send data as JSON
Data Ingest
Model
Create Model
Redis
Kicking off periodic
retraining
Save training data
Save model object Send JSON data
without label
Receive prediction from trained model
instance
Deployed at: http://dsoncf.cfapps.io
Code: https://github.com/pivotalsoftware/ds-cfpylearning
PREDICTION API ARCHITECTURE
$ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME
MODEL INTERFACE
Data Driven Applications
SIMPLE HTML + JS MODEL PREDICTIONS
http://ds-demo-transport.cfapps.io
RSHINY APP INTERACTIVE EXPLORATION
https://ak-insurance-demo.cfapps.io:4443/
Show off your data science related Cloud Foundry apps:
Twitter: @dsoncf http://dsoncf.com
@ianhuston @akagoshima
R E S T A P I
Send data as JSON
Data Ingest
Model
Create Model
Redis
Kicking off periodic
retraining
Save training data
Save model object
Send JSON data without label
Receive prediction from trained model
instance
Deployed at: http://dsoncf.cfapps.io
Code: https://github.com/pivotalsoftware/ds-cfpylearning
Visualization
PREDICTION API ARCHITECTURE
Data Services Bind and scale system services – Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME
Add User Provided Services – Standalone Hadoop or Apache Spark cluster, Big Data System $ cf cups SERVICE_INSTANCE -‐p "host, port, username, password"