Post on 01-Nov-2014
description
transcript
© 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.
Free Yourself with Cloud FoundryA Private Cloud Experience
Mike Heath, Shawn Nielsen, Mike Youngstrom
Disclaimer
This presentation does not represent the views, opinions,
policies, nor direction of The Church of Jesus Christ of
Latter-day Saints. These views are the sole views of the
presenters involved with the presentation. We take full
responsibility for content presented and any errors or
incorrect perception of representation.
Description / Overview
• This presentation is about
our experience integrating
Cloud Foundry into our
organization.
• It is not a tutorial on how
to deploy CF.
Presentation Roadmap
• Why we chose Cloud Foundry?
• Cloud Foundry APIs in a UI
• Gap analysis
• Develop and deploy
• Authentication & diagnostics
• Custom services
• Application level firewalls
Business Problems
• 100s of small/medium apps
• Difficult to manage infrastructure for so
many apps
• More interested in fault tolerance over
scale for most apps
• Slowness in provisioning time
The PaaS Team
• 2 ½ Developers
• ½ Manager
• 3 Operations personnel
• Passion for efficiency
Current State
• Currently in beta
– Moving to enterprise ready in Q4
• Good feedback so far
• Applications
– 4 production
– 14 development
Why we went with PaaS
• Developer productivity
• Fault tolerant and scalable
• Simplification of infrastructure
• Consistent deployments across runtimes
• Self-service
Why Cloud Foundry?
Other Reasons for Cloud Foundry
• Infrastructure agnostic
– Avoids vendor lock-in
• Public/private
• Cloud on cloud
– PaaS running on an IaaS cloud
Other Reasons We Love Cloud Foundry
• Open source
– Ability to adapt Cloud Foundry to our needs
– Community involvement
• Architecture
– Good architecture sells itself
Overview of Environment
• vSphere
• F5 (load balancer)
• Oracle
• LDAP
• Proxy based SSO
• Primarily Java+Spring shop
– Starting to see some NodeJS
Cloud Foundry Adoption Challenges
Cloud Foundry Adoption Challenges
• Convincing the systems engineers
– VM centric world to app centric world
Network Zoning and Firewalls
• Network firewalls
• Transition to host / app firewalls
Network Firewalls
Host / App Level Firewalls
Cost Challenges
• Current costs and billing models
– Multi-year bill-out
• How do you know what you’re going to use in 4 years?
You don’t!
Quota and Utilization Bill-back
Know your bill
$25 per GB RAM per month
Know your usage
Trade-in Model
Organizational Efficiencies
Presentation Roadmap
• Why we chose Cloud Foundry?
• Cloud Foundry APIs in a UI
• Gap analysis
• Develop and deploy
• Authentication & diagnostics
• Custom services
• Application level firewalls
Billing and Usage Through CC APIs
CloudController API:
url: "/v2/quota_definitions/36673f76-c617-4ae8-94b9-7adccb747ced”
entity: {
name: "Enterprise Management Organization - Quota“
non_basic_services_allowed: false
total_services: 100
memory_limit: 40960
trial_db_allowed: false
}
Custom UI
• Just like everyone else…we have our own UI.
• Why?
– No standard UI available. Though several community UIs on
their way.
– No way to do access management through CF. Why Not?
– Management of quotas for organizations
– Give management something visual to look at
Cloud Foundry APIs
CC URL: /v2/organization/<guid>
entity: {
name: “MySpringOrganization"
quota_definition_url: "/v2/quota_definitions/<guid>
spaces_url: "/v2/organizations/<guid>/spaces”
domains_url: "/v2/organizations/<guid>/domains”
users_url: "/v2/organizations/<guid>/users”
managers_url: "/v2/organizations/<guid>/managers”
auditors_url: "/v2/organizations/<guid>/auditors”
app_events_url: "/v2/organizations/<guid>/app_events”
}
s
Key Performance Indicator (KPI) Data
Quota
state: "RUNNING"
stats: {
uptime: 984643
mem_quota: 536870912
disk_quota: 1073741824
fds_quota: 16384
}
Live Usage
usage: {
time: "2013-09-10 02:05:27”
cpu: 0.0007454006633748
mem: 211116032
disk: 116498432
}
Cloud Foundry APIsCC URL: /v2/apps/<guid>/stats
Operations Center
UI Demo
Presentation Roadmap
• Why we chose Cloud Foundry?
• Cloud Foundry APIs in a UI
• Gap analysis
• Develop and deploy
• Authentication & diagnostics
• Custom services
• Application level firewalls
Gaps
Authentication
Diagnostics
Enterprise Services and Legacy
Presentation Roadmap
• Why we chose Cloud Foundry?
• Cloud Foundry APIs in a UI
• Gap analysis
• Develop and deploy
• Authentication & diagnostics
• Custom services
• Application level firewalls
Deployment
• Use BOSH– Fork cf-release
– Proxy?
• Environments:– X Dev
– 1 Test
– 1 Prod
• Customers only use prod
• Break deployments into pools– Core
– DEA/Router-X
Development
• Dev is prod but smaller– Use BOSH
– vSphere
– F5
– SSL – kind of
• Develop on component?– `bosh stop` existing component
– Configure local component in place
• Other options– Vagrant?
– Warden CPI?
Presentation Roadmap
• Why we chose Cloud Foundry?
• Cloud Foundry APIs in a UI
• Gap analysis
• Develop and deploy
• Authentication & diagnostics
• Custom services
• Application level firewalls
Customizing Authentication
• UAA & Login
– Java Spring + Spring Security
• Customized UAA
– Added UAA to src release
– Added spring config
• Login Server
– May re-visit
Buildpack Customization
• Vital extension point
• Support for legacy artifacts
• Just fork and tweak
– Proxies
– Oracle support for Node
– Pre-processed `npm install`
– Add JMX support
Problem: Diagnostics
• Need:– Thread dumps
– Heap dumps
– Remote debug
– APM
• Current story? Not good– Logging
– Console port???
– Debug port???
Solution: Caldecott?
• Use console and debug ports
• Caldecott
– TCP over HTTP proxy
– Inter-app communication
• Diagnostic
– Router/gateway/server
Solution: Diagnostics Server
Dev Workstation
Diagnostic Server
UAA/Cloud Controller
DEA
Warden
Request Proxy
Authorize Request
Proxy to DEA
Forward to Warden onDebug or Console Port
Demo Remote Debug
JMX
• JMX is great– Heap dumps
– Thread dumps
– Basic profiling
– Management operations
• JMX over RMI lacking– JMXMP to the rescue
• Configure JMX in buildpack– Bind to `console` port
• Created `cf` plugin to launch Visualvm
JMX Demo
Diagnostics Summary
• Caldecott
– Doesn’t work for us
• Diagnostic Server
• Safe customization?
– Removed ports
Presentation Roadmap
• Why we chose Cloud Foundry?
• Cloud Foundry APIs in a UI
• Gap analysis
• Develop and deploy
• Authentication & diagnostics
• Custom services
• Application level firewalls
Custom Services
• Integrate with existing systems
– LDAP, NFS/CIFS, Oracle, SSO,
Web Services, etc.
• New ‘user-provided’ service is not
adequate
– Will still work for most people
– We need more than a set of
key/value pairs
Custom Services Built on Java and Spring
• We’re a Java shop.
• Java is more enterprise
friendly.
– Client libraries for existing
systems
• Our team consists of
Java/Spring developers.
Custom Service Framework Features
• NATS Client
• Cloud Foundry utilities
– Type-safe NATS messaging
– PID file support for working with Monit/BOSH
– YAML config
• Cloud Controller client for invoking service APIs
• Service Broker
• Open source
– https://github.com/cloudfoundry-community/java-nats
– https://github.com/cloudfoundry-community/cf-java-component
NATS Java Client
• NATS is the distributed pub/sub messaging system used
by Cloud Foundry.
• NATS support is essential for customizing Cloud Foundry.
• Java client
– Built using Netty
– Integrates with Spring
• Generic NATS client, nothing Cloud Foundry specific
NATS Client Sample Code
// Connecting NATS to server
Nats nats = new NatsConnector()
.addHost("nats://localhost")
.connect();
// Simple subscriber
nats.subscribe("foo", (message) -> {
System.out.println("Received: " + message);
});
// Simple publisher
nats.publish("foo", "Hello world!");
NATS Client Spring Integration
<nats:nats>
<nats:url>nats://localhost:4222</nats:url>
</nats:nats>
@EnableNatsAnnotations
@Configuration
public class Configuration {
@Subscribe("foo")
public void onMessage(Message message) {
System.out.println(message);
}
}
Spring Boot/NATS Example
• NATS to HTTP Gateway
– Using Spring Boot
– Single Java source file
– Less-than 75 lines of Java (including import statements.)
https://github.com/mheath/spring-boot-nats-example
Cloud Foundry Specific NATS (cf-nats)
• Simplifies using NATS with Cloud Foundry
• Type-safe NATS messaging
– Component discovery
– Router registration
– Staging notifications
• Still a work in progress
https://github.com/cloudfoundry-community/cf-java-component/tree/master/cf-nats
PID File
• Support for creating .pid file at Spring context initialization
• Useful for working with Monit
– BOSH uses Monit
<cf:pid-file resource=“file:/var/run/component/my-cf-component.pid” />
YAML Configuration
• Existing Cloud Foundry components use YAML for
configuration.
• YAML is cool.
• We provide Spring integration for using YAML.
– XML configuration for loading YAML as properties.
– Java configuration for using YAML is a Spring
PropertySource.
Service Broker Framework
• Enables creating custom services
• Creating service broker requires implementing a single
Java interface
• Simple Cloud Controller client for invoking service APIs
• Provides APIs for automatically registering with Cloud
Controller
Example Service Broker
• Built on Spring Boot
• http://github.com/cloudfoundry-community/java-service-
broker-example
Example Service Broker Demo
Oracle Service Demo
Custom Oracle Service
• Prompts for: service name, schema/user, password
• Produces credentials:"credentials“ : {
"schema":"CF_DEV2",
"ldap“ : {"host":“fake-ldap.lds.org", "port":389, "context":"cn=OracleContext…"},
"firewall":[{"port":1234,"host":“oracle-scan-host”}…],
"descriptor":"(DESCRIPTION=(ADDRESS_LIST=…",
"alias":“DB-DEV2",
"service":“service134",
"addresses":[{"host":“oracle-scan-host","port":"1234"}],
"jdbcUrl":"jdbc:oracle:thin:@ldap://fake-ldap/DB-DEV2,cn=OracleContext,…",
"password":"super-secret-password"}}
Custom Service Creation Workflow
`cf` client
Service Broker
Cloud Controller
cf plugin prompts
for service data
cf plugin sends data
to broker (host:
‘foo’, port: 1234)
cf creates service
Cloud Controller tells broker
to create service instance.
Broker correlates service data
originally sent by cf plugin and
returns service credentials to
Cloud Controller.
Holds service data
(host: ‘foo’, port 1234)
Presentation Roadmap
• Why we chose Cloud Foundry?
• Cloud Foundry APIs in a UI
• Gap analysis
• Develop and deploy
• Authentication & diagnostics
• Custom services
• Application level firewalls
Firewall Requirements
• Proxy based SSO requires control over incoming requests
• Minimize attack vectors from compromised/malicious apps
– Limit access to console and debug ports
• Allow access to high risk network zones
• Auditability – What network resources does each app
have access to?
• Minimize customization of Cloud Foundry
Proxy Based Single Sign-On (SSO)
Warden
• Used by DEA to securely host applications
• Applications run with a “Warden Container”
• Warden Containers isolate resources– Network
– Disk
– Memory
• Think of a Warden Container as a “lightweight VM.”
Controlling Incoming Requests
• Load Balancer and Cloud Foundry routers easily secured.
• Securing requests to Warden Containers more difficult
– Need to allow incoming requests from routers
– Disallow outgoing requests directly to routers
• Cloud Foundry scales dynamically
– Routers can be added/removed from the system at any time
– Changing static configuration of all DEAs when router is
added or removed is unacceptable
Custom DEA Incoming Firewall
• Configure Warden to drop all incoming and outgoing
packets
• Customized DEA to track Cloud Foundry routers
– Routers already broadcast presence via NATS
– Use existing NATS messages to build router registry
• Warden uses `iptables` to isolate network resources
– `iptables` are Linux kernel level network rules
– Our firewalls piggybacks on Warden’s `iptables` rules
Custom DEA Outbound Firewall
• Service credentials
embed firewall rules
• Use Warden API to
open firewall holes
• Facilitated by custom
services
"credentials" : {
...
"firewall" : [{
"network" : "10.118.50.0/24",
"port" : 8080
}]
}
Securing Outbound HTTP
• Problems– Single IP address can host
multiple web sites/applications
– Public web services often have large pool of IP addresses that can change over time
• Solution – HTTP proxy for hostname validation
• Custom service firewall rules enables access to the HTTP proxy
"credentials" : {..."firewall" : [{“http" : “http://www.google.com"}]}
Http Proxy Firewall Workflow
HTTP Proxy
DEA
Warden Proxies HTTP requests
Network Firewall
Intranet
Internet
Summary
• Cloud Foundry is awesome!
• Cloud Controller APIs make creating a UI easy.
• Cloud Foundry authorization customization is maturing.
• Diagnostics need work. Our solution was promising.
• You can create services in Java.
• We isolate our applications in app firewalls
Questions?
Citation references• Cloud picture (slide 3) Copyright 2007, Karin Dalziel. Creative Commons licensed.
• Snail picture (slide 4) Copyright 2013, Eirien (Ilweranta). Create Commons licensed.
• Wrench photo (slide 5) Copyright 2009, zzpza. Creative Commons licensed.
• Building photo (slide 10) Copyright 2010, Ricardo Diaz. Creative Commons licensed.
• Lightening road screen do not enter picture (slide 12) Copyright 2008, Bobby. Creative Commons licensed.
• Gap (slide 28) Copyright 2013, Upupa4me. Creative Commons licensed.
• “Key Note” (slide 29) Copyright 2009, William Neuheisel. Creative Commons licensed.
• “Angry Computer Support Worker Baning His Fists On His Desk” (slide 30) Copyright 2010, jfcherry. Creative Commons licensed.
• “24 Hour Service” (slide 31) Copyright 2008, Natalie Maynor. Creative Commons licensed.
• “Launch of Atlas V TDRS-K from Cape Canaveral AFS” (slide 32) Copyright 2013 NASA Goddard Space Flight Center. Creative Commons licsened.
• “American Architect, 1909” (Slide 33) Copyright 2011 REVIVALthedigest. Creative Commons licensed.
• “Keys” (Slide 34) Copyright 2005 mmarchin. Create Commons licensed.
• “Lego building” (Slide 35) Copyright 2006 Matt Bateman. Creative Commons licensed.
• “Heavy Metal: TDK MA-R90 Cassette Tape” (Slide 39) Copyright 2012 Scott Schiller. Creative Commons licensed.
• “Tunnel” (Slide 41) Copyrigh 2010 Dushan Hanuska. Creative Commons licensed.
• “Roadside Service” (Slide 42) Copyright 2012 Pam Morris. Creative Commons licensed.
• “Typical Prison Guard” (Slide 60) Copyright 2009 Son of Groucho. Creative Commons licensed.