Copyright © 2014 Splunk Inc.
Michael de Buin, Schuberg Philis [email protected] Gert Kremer, Schuberg Philis [email protected] Dani Flexer, Splunk [email protected]
Islands of Splunk MulJple Splunk as a Service Architecture and ImplementaJon
Disclaimer
2
During the course of this presentaJon, we may make forward looking statements regarding future events or the expected performance of the company. We cauJon you that such statements reflect our current expectaJons and
esJmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,
please review our filings with the SEC. The forward-‐looking statements made in the this presentaJon are being made as of the Jme and date of its live presentaJon. If reviewed aTer its live presentaJon, this presentaJon may not contain current or accurate informaJon. We do not assume any obligaJon to update any forward looking statements we may make. In addiJon, any informaJon about our roadmap outlines our general product direcJon and is subject to change at any Jme without noJce. It is for informaJonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaJon either to develop the features or funcJonality described or to
include any such feature or funcJonality in a future release.
Agenda
! The MulJ-‐Splunk-‐as-‐a-‐Service (MSaaS) framework ! MSaaS implementaJon @ Schuberg Philis ! DemonstraJon
3
MSaaS Architecture
Background ! Splunk administrators are increasingly required to provision Splunk as a service offering to mulJple customers – Commonly requires provisioning a Splunk instance to each customer
! MSaaS is a conceptual framework designed to help deliver such an offering
! Schuberg-‐Philis have implemented this framework using Chef on Apache CloudStack
5
FuncJonal Requirements
6
! AutomaJc deployment and update of mulJple Splunk instances ! Packaged binaries, scripts and configuraJons ! Modular ! Each instance can scale from very small to as large as necessary ! Each instance is customized as needed ! System must funcJon independently without external resources
– Except for authenJcaJon, a datacentre automaJon (DCA) tool and an opJonal license manager to fulfill its purpose
! Archiving, backup and resilience requirements are defined per customer
FuncJonal Requirements ! Resilience
– Disaster recovery (DR), no single point of failure, indexing Jer resilience, storage resilience
! Strict data segregaJon ! Different network jurisdicJons isolated from each other with no shared resources nor shared informaJon
! Cross-‐jurisdicJon access possible when explicitly enabled ! JurisdicJon hierarchy supported
– A jurisdicJon can include other jurisdicJons
7
FuncJonal Requirements ! Indexed data can be shared, subject to jurisdicJon, and indexes copied between instances
! MulJple licensing models supported – central license manager and license pools – per-‐Island license manager and mulJple keys assigned on deployment – global license
! Roles defined independently for each instance ! All credenJals maintained in the enterprise idenJty management system and allocated at deployment
8
Architecture Concepts: Island ! A complete Splunk deployment ! No informaJon sharing between Islands ! Own set of users and roles ! Manages a set of Forwarders ! Forwarders can send data to many Islands but are managed by a single Island
9
Architecture Concepts ! Bridges
– Islands without indexing capabiliJes that enable search on mulJple Islands
! Deployment unit – An independently deployed collecJon of Splunk
components ! Customer
– An independent user of Splunk — a business unit or customer
! Island service agributes – ReplicaJon factor, search factor, DR requirements,
security, backup, storage Jer, “performance”, retenJon plan, daily volume
! AdministraJon Island monitors the other Islands
10
MSaaS Deployment Server ! A centralized system responsible for installing, configuring, and updaJng the Islands
! Maintains all binaries, applicaJons, configuraJons, apps and rouJng informaJon in a version control system (VCS)
! Updates the Islands’ binaries when necessary
! Each Splunk DS maintains its Island’s components – ConfiguraJon files supplied by the MSaaS
DS and propagated by Island DS
11
ApplicaJons • Each Island has any number of Splunk applicaJons — a.k.a. apps • Apps are
– Managed centrally and deployed with the Island – Versions are maintained in the VCS – Customized for the MSaaS as a whole or for a subset of the Islands
! MSaaS administraJon apps reside on a dedicated Island – Monitors the other Islands for usage, security, charge-‐back, health and need
for maintenance
! Standard apps deployed with each Island – S.o.S — Splunk on Splunk – App for Unix or for Windows Infrastructure as appropriate
12
MSaaS ImplementaJon @SBP
My Company and I Our customers:
Gert Kremer Mission CriJcal Engineer since 2007
Engineering the MSaaS Architecture
15
! Dual datacenter setup, no data backups (RF=SF=2) ! Maximum 100 GB/day per Island ! Centralized license server ! AcJve-‐standby search heads (rsync-‐ed)
! Not implemented: – Splunk Deployment Server – AdministraJon Island – Dedicated Job Servers
Ingredients
16
Requirements & use cases M-‐SaaS Meta Architecture DescripJon
Believers
Splunk Chef Cloud(Stack) Github enterprise
Will it blend?
17
Configura)on Indexing: Avg KBPS
Indexing: Avg EPS
Search: Avg First Event (sec)
Search: Avg Search (sec)
HP DL380G7; CPU: 2×6 Xeon 2.67GHz; Memory: 12GB; OS: Linux 64-‐bit, Fedora 14 (*)
22,400 79,057 2.48 20.18
Linux on EC2: c1.xlarge 800 pIOPS (*)
12,410 43,639 2.12 27.37
SBP: 4 CPU/16GB 12,449 43,865 2.82 18.24
SBP: 8 CPU/32GB 14,715 51,959 1.37 17.24
(*) hgp://blogs.splunk.com/2013/06/06/splunkit-‐v2-‐0-‐2-‐results-‐ec2-‐storage-‐comparisons/
Prior art on Splunk, Chef and Cloud
18
! Best Buy Splunk cookbook hgps://github.com/bestbuycom/splunk_cookbook
! OpsCode Splunk cookbook hgps://github.com/opscode-‐cookbooks/chef-‐splunk
! Splunk Storm (Splunk as a Service) hgp://www.getchef.com/customers/splunk/
Scope of AutomaJon
19
! Island creaJon ! Server instanJaJon ! Splunk soTware installaJon
! Cluster configuraJon ! Data disk and indexes: creaJon and management
! Data replicaJon between Search Heads ! Security (firewall rules, SSL setup) ! Monitoring (process, connecJvity, cluster health, Splunk alerts)
! Single-‐sign-‐on (SSO) ! Splunk applicaJons
Splunk Enterprise at Schuberg Philis
Search head Search head
Indexer Indexer Cluster master
License server
Datacenter 1
Datacenter 2
SupporJng Systems
Splunk Enterprise at Schuberg Philis
Bridge (SIEM)
Search head Search head AcJve Directory
License server
SupporJng Cloud
Infrastructure
AuthenJcaJng proxy
Island Customer …
Search head Search head
Indexer Indexer Cluster master
Island Customer …
Search head Search head
Indexer Indexer Cluster master
Island Customer …
Search head Search head
Indexer Indexer Cluster master
License Server
Island InstanJaJon
Search Head
Indexer Cluster Master
Proxy Servers
Indexer
Search Head
Datacenter 1
Datacenter 2
1. Configure island 2. Deploy island 3. Integrate island
1. Configure island 2. Deploy island 1. Configure island
License Server
Island InstanJaJon
Search Head
Indexer Cluster Master
Proxy Servers
Indexer
Search Head
Datacenter 1
Datacenter 2
1. Configure island 2. Deploy island 3. Integrate island
1. Configure island 2. Deploy island 1. Configure island
Returning the favor
24
! Generalize available Chef Cookbooks ! Splunk monitoring Nagios plugin available in Nagios Exchange ! Splunk deployment best pracJces and tools
DemonstraJon by Michael
Process Requirements
27
! Service Request – When customers request the service, a process is triggered that results in a
deployed instance of Splunk that implements the customer’s use-‐cases and the other agributes of the service requested.
! Charge Back – The cost of the service to its operators can be charged back to its customers
based on the actual cost of provisioning the service.
! Easy to Onboard – The process of incorporaJng data sources into the system is well defined
and simple
Data RouJng ! Islands use-‐cases can overlap requiring them to share data and data-‐sources
! Data rouJng is maintained in a global rouJng table ! On update, the rouJng table is converted into Splunk configuraJon elements suitable for inserJon into the transforms.conf file that is then propagated to the data collecJon Jer by the Master Deployment Server and Island Deployment Servers
! Heavy Forwarders are required to support data rouJng, otherwise a Universal Forwarder is used
Island Service Agributes ! Agributes are defined when a service is requested by a customer ! The provisioning process implements the agributes and deploys the Island
! Agributes – Resilience – DR – License volume – Data segregaJon and availability – Storage volume – RetenJon requirements – Data sources and use cases
Customers ! Each customer is an area of the enterprise or an external organizaJon that requires Splunk
! A customer consists of one or more use-‐cases ! MulJple customers can be consolidated into a single MSaaS Island segregaJon requirements permizng
Source Types and Data Sources ! In addiJon to the default set of source types provided by Splunk, MSaaS implements addiJonal source types that are used as needed by any Island
! When an Island is implemented, new source types are defined as required and the pre-‐defined MSaaS source types extended
! Data sources indexed in a given Island are parJJoned into specific indexes based on security, ownership, visibility constraints, retenJon requirements, and resilience requirements.
! These agributes can differ between Islands for common data sources, depending on the requirements of the use-‐cases implemented by this Island
Common Namespace and InformaJon Model
! Named Splunk enJJes are named in a common namespace so different Islands can share informaJon, apps, indexes etc.
! Requires MSaaS to maintain a centrally controlled global name space
