Rh summit2015 presentation_v2.5

transcript

Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation's express consent.

Openstack | Openshift| FICOLesson learned over the past 12 months

May 22, 2015

Agenda

»Who is FICO?

»Why Openstack

»FAC (FICO Analytic Cloud)

»Why Openshift

»Architecture

»Automation

»Questions

Who is FICO?

»FICO Score, MyFICO» Analytics software and tools for businesses

» Debt management

» Fraud & security analytics

» Customer engagement

» Big Data Analytics

» Designs custom algorithms for many specific use cases

» 90% of all lending decisions in the U.S. rely on the FICO Score

» 2.5B credit cards globally are protected by FICO Fraud Systems

» Founded 1956

» 50+ year history of data and analytics excellence

“The FICO Analytic Cloud provides Platform-as-a-Service (PaaS) access to the FICO Decision Management Platform, allowing customers to use FICO tools and technology to create and deploy applications and services.” – Tony McGivern, FICO CIO

Why move from VMware to Open Source?

Why move to the Cloud?

»Need to expand beyond top tier financial services companies and get into middle market

»Traditional on premise technologies moving to a self service or XaaS/SPI model

»Open Source nature allows for participation in driving change

»Need to lower cost

»Thousands of virtual and container instances

»Faster time to market

»Templatize apps deployment regardless of public cloud, private cloud, or underlying infrastructure

OpenStack Infrastructure

FICO Cloud Architecture v1.0

» Cisco UCS Converge Infrastructure

FICO Cloud Architecture v1.0Lessons Learned

» Complex Design» Too many technologies and distributions to select from» Difficult to troubleshoot when things go wrong» Lack of mature tools for capacity planning» Poor “end to end” integration between IaaS, PaaS, XaaS» No true documented architecture designs and few white papers

made public for customers/users

» FICO Cloud v1.0» Red Hat Enterprise Virtualization and Red Hat OpenStack» RHEL 6.x» Optimize design for compute workloads» Gluster used for primary and only shared storage solution» OpenShift used for PaaS

HAProxy Load Balancer

Openstack Controller/CEPH

OpenStack Compute/CEPH

» Cisco UCS C240M3 & C220M3

C220 C240 C240

» Simplify design» OpenStack (KVM)» CEPH for primary and only distributed shared storage» RHEL 7.x» Simplify design for easier handoff for Operations teams

» Targeted baseline optimized for compute and storage performance

» Haproxy / Keepalived

» Cloudforms Deployment/Proof of Concept

» Neutron SDN» Layer 2 only

» Performance issues and realizations» Does Ceph work for all workloads?

» Does the HW selected fit the application and product requirements?

Openstack Controller Node

OpenStack Compute/CEPH Node

» Cisco UCS C240M3 & C220M3

SolidFire SF4805

F5 Load balancer

» F5 to load balance between Openstack APIs

» Selected the Cisco C240 rack servers as the standard hardware “workhorse” to provide compute and storage

» Moved the OpenStack controller layer to Cisco C220 rack servers

» Implemented tiered Storage» Ceph to provide non SLA driven storage» SolidFire distributed flash storage to provide high performance storage

» Revised the sizing of our OpenShift Node

» Cloudforms single pane of glass for private and public cloud

» Swift object oriented storage

OpenShift

Why Openshift?

» On-premise

» Business needs to reduce the Time to Value and Time to Market

» Templatize apps deployment regardless of public cloud, private cloud, or underlying infrastructure

» Ability to build custom Cartridges

» Decrease development costs & simplified tools used for development and deployment

» FICO has reduced time to value for developing analytic solutions by up to 70% versus legacy on premise software

FICO Analytic Cloud | Openshift

» Launched FAC in 2013 before containers were cool

» Business needs required rapid deployment and continuous integration while simplifying the current design

» FICO Analytic Cloud provides IaaS and PaaS access to the FICO Decision Management Platform for multiple tenants and customers

» FICO Decision Management Analytic Platform allows application developers, business users and FICO partners around the world “one-stop” access to use FICO technology to create, customize, and deploy analytics-driven applications and services empowering them to make educated decisions and lowering risk

» FAC/DMP was built using Openshift as the backend engine

Decision Management Suites

» Platform as a server that allows non-developers(e.g. data scientists) to construct and analytic solution through the use of a rapid application development and catalog of FICO component

» User “stiches” together FICO components with appropriate data flow and business rules RAD environment

» Solution is then deployed via Openshift Enterprise for consumption

» Full lifecycle management allows user to develop promote to a separate test environment and ultimately promote to run time / production

» OSE not used for internal DevOps, but instead is a productization for customer consumption

FICO Analytic Cloud Architecture | Openshift v 1.0

Openshift | 1.0

» Installation was manual (bind, activemq, mongo,OSE)

» Deployed on existing VMware infrastructure & Netapp

» Limited online documentation

» Single Point of Failure (1 broker/dns/activemq/mongodb)

» Break fix / Bugs

» Tips:

» Documentation

» Operationalize Openshift

Openshift | 1.1

» Node settings (base on your environment Application/node size

» Gear Idling (2hrs)

» Node gear size/profiles

» Over commitment: number of active gears per node

» Identify & eliminate SPOF

» Openshift (broker, bind, activemq, mongo)

» Application ==> scaled apps

» Tips:

» Monitoring

» DNS

» Platform (internal/external name resolution)

» Offload name resolution Infoblox DNS Appliance

» Dynamic DNS on Bind

Openshift | 1.2

» Openshift Upgrade

» OSE Deployment via PXE

» HA design (3 brokers, 3 mongo, 3 activeMQ)

» F5 (brokers)

» Tuning Openshift

» Load Test (jmeter / custom scripts)

» ActiveMQ JVM OOM (activemq.xml)

» MongoDB(set primary)

» Tips:

» Monitoring / Trending (Keynote, Zabbix, Cacti)

» Total active gears

» Total Capacity per environment

» Total idled gear in the past 90 days

Openshift | Monitoring | Trending Stats

Openshift | Current Architecture

Openshift | 2.0

» New deployment

» Parallel 1.2 / 2.0

» Openstack / CEPH

» Foreman / Heat OSE Deployment

» Implemented active/passive DR/HA across 2 datacenter to meet SLAs

» OSE 2.2 , 3.0 origin

Automation

Our Tools Foreman, Puppet, Cisco UCS

Provision

Configure

Manage

Why FICO Automated Using Puppet and Foreman

» Save Time and Manpower.

» Ensured consistency of configurations across all nodes.

» Repeatability of deployments.

» Source controlled configurations.

Foreman

» Foreman can deploy to both virtual and bare metal machines.

» Foreman Provisions RHEL on Cisco UCS C240s and C220s

» Foreman Host groups with Parameter overrides preps the Cisco UCS 240s and 220s for puppet configuration.

» Foreman Proxies in every Geo-location allows ease of deployment.

Puppet

» FICO Custom Puppet Module to install OpenStack Icehouse and Juno.

» Custom Puppet module was built using many different Puppet Forge Modules.

» Wrote custom facts to tie everything together.

» Reduce build time from days to minutes. » Average build time per server is 3 minutes.

Satellite 6

» Consistency of Packages across OpenStack, Openshift and VMware

» Repeatability of deployments

» Controlled package repositories.

» Different views per release, allowing testing without affecting current environments.

» Reduce the amount of tools within our environment

» Already using components of Satellite 6» Foreman» Puppet

» Better integration with CloudForms than Spacewalk

CloudForms

» Manage OpenStack and legacy VMware from one tool

» Resource usage and optimization

» Self-service portal and catalog

» Single API and GUI for both development and production environments

» Dashboards and reporting

» Ahead of the curve on integration» Professional services needs» Needed functionality 1 – 2 releases out

Red Hat Ceph Storage

» Why Ceph?» Scalable» Open Source» Software-defined storage» Provides both block and object oriented storage

» Optimize small footprint deployments with OpenStack/CEPH converged hardware

» Tight integration with OpenStack

» SolidFire for high performance SLA driven workloads» Scalable, clustered all-flash storage array provides inline compression,

deduplication and replication

Red Hat Enterprise Linux

» RHEL 6 & 7» OSP runs on RHEL 7

» Provides solid platform understood by Operations

» Common and supported implementation across multiple products» Cisco UCS» Oracle RAC

What’s Next

» Use UCS Central automation for quick stand up of private cloud environments regardless of geo location or availability zone

» OpenStack Kilo Automation» Puppet OpenStack module?

» Cisco UCS M4 series

» Eliminate Ceph and use only Solidfire distributed flash storage?

» Migrate away from NFS/NAS to object oriented storage such as Swift?

» Satellite 6 replacement for Spacewalk, etc

» CloudForms» Single pane of glass» Ability to manage both Private

and Public Clouds» Capacity planning» Reporting» Integration with OpenStack and

OpenShift» Single API regardless of the

endpoint

Questions ?

Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation's express consent.

Thank You

May 18, 2015

Nick GerasimatosNickGerasimatos@fico.com

Chris FerraroChrisFerraro@fico.com

Kiet Quangkietquang@fico.com

Rh summit2015 presentation_v2.5

Recruiting & HR