Agenda
Why introduce Run Book Automation
Planning
Tool selection and appraisal
Experience so far
RBA Design Framework
Summary
30.09.2011 2 Titel der Präsentation und Name des Redners
RBA implementation project objective
1. Main objective: Improve RUN efficiency in Infrastructure Operations – reduce cost
2. Develop a RBA platform for all global data centers
3. Establish a RBA service
Quality criteria
Improve provisioning speed and quality
Make automation available to all services
Ensure reusability of flows and low maintenance effort
Quickly deliver financial benefits
Oh, almost forgot: Make this happen in a infrastructure that was
build over the last 10 years without automation in mind…
30.09.2011 4 Titel der Präsentation und Name des Redners
Vision: Fully automated provisioning – no hands-on tasks
Rapid Provisioning
High Standardization
Up-to-date Documentation
Reliable Quality – No Human Errors
Low Process Costs
No Workload Buildup
Less Operational Management
30.09.2011 5 Titel der Präsentation und Name des Redners
A
u
t
o
m
a
t
i
o
n
Manual Provisioning
Industrialized Provisioning
Fixed Service Offering
Self-service Portal with service request dialog
Detailed execution logs
Reliable lead time and quality
Planning
30.09.2011 7 Titel der Präsentation und Name des Redners
POC for
tool selection
RBA Design Framework Iterations
Pilot
Virtual Windows Server
Establish RBA Service
2010 2011 2012
Other Flows and Automations
(HP SM Interface, etc.)
Management
Decision
More Automations in
Munich
Automations for
global
Datacenters
Re
vie
w
Automations in the making
Installation of Managed virtual Linux Server
Installation of Oracle Databases
Promote AD Member Server to Domain
Controller
Automation for HVD data store administration
(later reuse for Netapp Storage London)
HP OO wins in proof of concept
3 tools in POC
5 automation tasks to
demonstrate capabilities
Only HP OO can complete all 5
tasks within a set time
9
Category HP OO B C
Accomplishment of mandatory workflows ++ ++ +
Accomplishment of optional workflow ++ o o
Skills of personnel (Number of personnel) ++ ++ +
Number of pre-assembled connectors and potential
workflow development effort +++ + o
Overall quality of results ++ ++ +
Pricing + + -
HP OO tool appraisal after nine month of flow
customizing
30.09.2011 10 Titel der Präsentation und Name des Redners
In general very happy with HP OO tool!!!
Easy to use
Flow customizing more efficient than classical scripting
Program control in graphical flows is easy to understand
HP OO Central Web Portal
More functionality for OO Central user input dialog
Calendar for date selection, set default values, radio button, etc.
Fix one or two bugs OO Central
HP OO Studio features to improve
Flow synchronization and staging is complex
Replace file system based repositories with database repository with staging
functionality
Coordination of shared flow customizing is difficult
Improve check-in/check-out functionality
Feature to synchronize system accounts between repositories
More transparency on flow permissions, better handling
Non-functional
Further development internet community and knowledge base
Optimize support reaction times
Experienced HP OO experts in short supply
Accelerator Packs and Integrations are like coding examples but no simple drag&drop
People
People like their jobs – sometimes not readily offering information/support
For automation we need to now how things are done
but also why they are done this way – understanding
both takes time
Automation requires cross-service collaboration
Initial concern about automation going berserk
In the beginning people were critical – my job is to difficult for automation
Automation without peoples support may be possible but effort explodes
30.09.2011 12 Stefan Zahrer
Technology
Infrastructure is complex – very, very complex indeed
Ask for picture of the architecture – discover it’s in
people’s heads
Work on production systems as early as possible – lab
systems are not the same
Architecture must be designed with automation in mind
Automation can be driver for standardization
30.09.2011 13 Stefan Zahrer
Run Book Automation is organizational change for all
IT functions
30.09.2011 14 Titel der Präsentation und Name des Redners
Automation Benefits
High Standardization
Fixed Service Offering
Fast and reliable
provisioning
Infrastructure Operations
Design overarching
processes and consistent
infrastructure
Cross-service collaboration
Work on architecture, not in
run operations
Share responsibility for
automated service delivery
Outsourcing Provider
Collaborate in automation
design, build and release
Include automation in
continuous service
improvement process
Support operators to
become experts with cross-
service knowledge
Application Development
Identical processes for all
projects
Develop Self-Service
provisioning know-how
Standardize architecture to
a higher level
Technical Infrastructure
Standardization on all
levels (Storage, Server,
OS, DB, …)
Administration designed for
automation (tool-based
decisions, etc.)
Offshoring
More advanced
administration tasks
Better understanding of
infrastructure required
Participation in automation
design
Documentation
Documentation in
automation tool
Consistent and reliable
Focus on technical
administration processes
Develop
People
Develop
Business
Model
Develop
Standards Develop Qualification
The Gordian Knot – how to ensure reusability and
maintainability
HP OO flows execute functions in infrastructure systems based on organizational processes
Process sequence and technical operations are closely interwoven
What happens if things were to change?
In the infrastructure: new software version, feature changes, new systems, …
In the organization: Changes to processes, new CMDB attributes, …
30.09.2011 16 Titel der Präsentation und Name des Redners
Automation Flow
Automation Flow
Automation Flow
Automation Flow
Infrastructure
Major topics of RBA Design Framework
30.09.2011 17 Titel der Präsentation und Name des Redners
Flow Authoring
Flow Architecture
Flow Call Relation-
ships
Naming Conventions
Authorization Model
ITIL Integration
Change Management
Incident Management
CMDB Maintenance
Downtime Management
Freeze times, veto, …
Flow Staging and Deployment
Flow Repositories
Flow Release Process
Private Database
Advanced logging / reporting
Documentation of user input
Infrastructure Topology Model
Infrastructure Interface (II)
Action Plan (AP)
Optional: Subcontroller (SC)
Controller (CO)
User Interface (UI)
Utility
(U
T)
Transactional Data
Persistence Table
Key Value Store
Logical
Logging
Server Name
Table
Master Data Infrastructure Topology
Model
Select and Start User-
Interface-Flow in HP OO
Central
Input in User-Interface-
Flow of HP OO
System Call to I3S
(Create/Save Change)
System Call to I3S to
verify Change
Relevance
Change Relevant
Flag set?
Yes
Input Risk- and Impact-Parameters
Execution Time?
Start immediately after aproval
No
Submit Change
Input Parameters for Downtime
Request
Downtime Window
Input Parameters for
Time is agreed (Start
Time/Finish Time)
Fetch Downtimes via I3S for first
task
Time is agreed
Possibility that
downtime window
is not available
Writing of Variables in Private
Database
End of User Interface Flow
Aproval Process
First Automated Task (Status assigned)
I3S creates XML-File
Scheduled File Check picking up XML-File
Start of Scheduling Flow
- Acceptance of I3S Task
- Searching for Downtime of every CI in I3S
- Scheduling of Controller Flow for every CI at defined
Downtime or execute immediately
Start
Start of scheduled Controller Flow
Check if respective Change is canceled
No
Canceled? YesAbort
Report in Private Database
CI still exists in I3S
Change?
No
Yes
Check if running in
Downtime Window
Mode (Flow Input)
Check if Downtime is
still valid
Yes
No
Yes
Read Input Values from Private
Database
Do Work on CI
Exited with Error?
Update Data in Private DB
Check for last CI
Document in I3S
no
Create Incidentyes
Close Task
yes
Exit
no
Summary
Tools are (finally) ready for large style automation
Vendors starting to include interfaces for automation
in their products
Automation of existing infrastructure is possible
to large extend
Long time risk in using tools without
architecture model in place
Implementation effort is significant
Financial benefit/ROI depends on number of runs
30.09.2011 19 Stefan Zahrer
Flow Architecture – separates process control and
system control
30.09.2011 22 Titel der Präsentation und Name des Redners
Infrastructure Interface (II)
Action Plan (AP)
Optional: Subcontroller (SC)
Controller (CO)
User Interface (UI)
Utilit
y (
UT
)
Creates GUI for users
• Populates selection lists
• Verifies user input
• Checks user permissions
Controls processing order
of tasks
Executes a single function on
one infrastructure system
Combines functions to execute
a complex task on one system
Combines tasks on different
systems to reduce complexity
of Controllers
Characteristics of flow types
30.09.2011 23 Titel der Präsentation und Name des Redners
Flow Type Complexity Reusability Modify and Test for
change to
Modify/Test
Effort
User Interface Very High Very Low - User input
- Infrastructure topology
- Organizational Process
Can be tested without affecting
infrastructure
Medium
Controller High Very Low Organizational Process High
Subcontroller Medium Low Technical Process High
Action Plan Medium High Infrastructure systems and
applications
Low
Infrastructure Interface Low High Infrastructure systems and
applications
Low
Utility Low High New HP OO versions only Low
Rules for flow calls limit complexity
30.09.2011 24 Titel der Präsentation und Name des Redners
Infrastructure Interface (II)
Optional: Subcontroller (SC)
Controller (CO)
User Interface (UI)
Utilit
y (
UT
)
Action Plan (AP)
Execute Configuration Changes
Infrastructure Interface (II)
Execute Configuration Changes
Action Plan (AP)
Infrastructure Systems Private DB
Logging, etc.
Scheduler Persistence
Table
• Gather information
• Create i3S Changes
• Logging
Private Database enhances HP OO standard features
Reporting of success/failure
Detailed reporting for services
Documentation of user input
Handover variables for scheduled runs
30.09.2011 25 Titel der Präsentation und Name des Redners
Transactional Data
Persistence Table Flow run specific variables
Key Value Store Global Constants
Logical Logging Basis for result reporting
IP-NameTable Server Name Mgmt.
more to come…
Master Data
Infrastructure
Topology Model
Infrastructure Topology Model
Dependencies in infrastructure (Locations, Networks, AD Entities, SW Packages, order, preferences, etc.)
Supports guided user input
Reduces flow complexity
Infrastructure Topology Model stores information in a
structured easy to maintain form
Main purposes
Enhance User Input flows
Content of list boxes, default selections
Limit selections to valid combinations
Dynamic selections in Controller flows based on
System architecture, location, usage, …
Preferences (e.g. prefer one ESX Cluster
whenever possible)
Control software installation flows
Software versions/editions
Installation order
Required and optional packages
Staging data from DEV to PRD
30.09.2011 26 Titel der Präsentation und Name des Redners
RBA_Types
ID
Type
Description
int
varchar(512)
varchar(512)
<pk>
RBA_Entities
ID
Type_ID
Name
Description
int
int
varchar(512)
varchar(512)
<pk>
<fk>
RBA_Entities_RBA_Entities
ID
Parent_Entity_ID
Child_Entity_ID Description
int
int
int varchar(512)
<pk>
<fk2>
<fk1>
RBA_Entities_Extended_Attributes
ID
Entity_ID
Name Value
Description
int
int
varchar(512) varchar(512)
varchar(512)
<pk>
<fk>
RBA_Entities_RBA_Entities_Extended_Attributes
ID
Entitiy_Entity_ID Name
Value
Description
int
int varchar(512)
varchar(512)
varchar(512)
<pk>
<fk>
RBA_Entities_RBA_Entities_RBA_Entities
ID
Entity_ID
Entity_Entity_ID
Description
int
int
int
varchar(512)
<pk>
<fk1>
<fk2>
RBA_Entities_RBA_Entities_RBA_Entities_Extended_Attributes
ID
Entitiy_Entitiy_Entity_ID Name
Value
Description
int
int varchar(512)
varchar(512)
varchar(512)
<pk>
<fk>
No “hard encoded” values in flows
Reduce flow complexity – data sets defined by SQL queries
Less need to change to flows – edit database or SQL statement only
Database changes are immediately available in all affected flows
Execution Flows User Interface Flows
Private DB
Interaction between flows and Private DB
30.09.2011 27 Titel der Präsentation und Name des Redners
Dialog Step 1 Dialog Step 2 Summary
Step
SQL SQL SQL
HP OO
Scheduler
Controller Flow
Infrastructure Topology Model
Persistence
Table
Logical
Logging
Table
SQL
Dependencies virt. Windows Installation – simple view
30.09.2011 28 Titel der Präsentation und Name des Redners
Location
Name
Hosting
Site
Vmware
Cluster
Windows
Server OS
Version
Windows
Server
Packages
Windows
Server
Role
AD
Domains
Type
Vmware
Network
Label
Sites Usage
Cost
Center
Windows
Server
Add.
Groups
Network
Segments
Vmware
Data-
center
Windows
SW Depot
Vmware
vCenter
Windows
Server OS
Edition
Windows
Server
Image
Network
DNS
Domains
Vmware
Resource
Pool
Vmware
Datastore
Suffix
Hostname
Prefix
Owner
Group
Vmware
Datastore
Exclude
Backup
Environ-
ment
Job Mgmt
Server
Most input is selected
from list boxes
Comments support user
Input dialog with list boxes – supports the user input
and avoids typos
30.09.2011 30 Titel der Präsentation und Name des Redners
Step by Step
Input
Step 1 Step 2 Step 5: Input Summary
Only valid combinations can be selected – less room
for errors
30.09.2011 31 Titel der Präsentation und Name des Redners
Step 1 Step 2
Limited
selections
Infrastructure
Topology Model
Cost Center and description
from database
Detailed logical reporting of all changes to
infrastructure
30.09.2011 32 Titel der Präsentation und Name des Redners
UUID = 7a18cc03-5cba-4d2b-9eab-1b594f3b38bc
Timestamp RunID ChangeID Requester CI Status Flow Comment
7/21/2011 2:27:50 PM -30 C1234567 n059307 Start CO.windows.vm.create Start of VM Create
7/21/2011 2:27:50 PM -30 C1234567 n059307 Start AP.infoblox.getNewHostname
7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Success AP.infoblox.getNewHostname
7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start AP.infoblox.getIpByName
7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Success AP.infoblox.getIpByName
7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Success AP.dtm.addDownTimeWindow
7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start SC.vmware4.vm.deployHost Start of VM Deploy
7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.vm.getDetails
7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 VM not found AP.vmware4.vm.getDetails
7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.customizationspec.modify
7/21/2011 2:27:52 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.customizationspec.modify
7/21/2011 2:27:52 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.datastore.getLargest
7/21/2011 2:27:54 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.datastore.getLargest
7/21/2011 2:27:54 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.vm.deploy
7/21/2011 2:27:55 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.vm.deploy
7/21/2011 2:27:55 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.task.waitUntilFinished
7/21/2011 2:46:34 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.task.waitUntilFinished
7/21/2011 2:46:34 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.vm.setCpuAndMemorySize
7/21/2011 2:46:49 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.vm.setCpuAndMemorySize
7/21/2011 2:46:49 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.vm.setPowerState
7/21/2011 2:46:59 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.vm.setPowerState
7/21/2011 3:05:18 PM -30 C1234567 n059307 mucs801126 Failure SC.vmware4.vm.deployHost Windows Deployment failed!
7/21/2011 3:05:18 PM -30 C1234567 n059307 mucs801126 Failure CO.windows.vm.create ServerDeploy failed!
HP OO executes functions on infrastructure
components
30.09.2011 33 Titel der Präsentation und Name des Redners
HP OO i3S CMDB
DPPM
Infoblox
Network
Networker
Backup
$Univers
Job Mgmt AD
Directory Service
DTM OVO
Monitoring
SCOM
Monitoring
i3S Incident
i3S Change
SCCM
VMware
vSphere
Netapp
Storage & Fabric
Windows / Linux
OS
HP SA
HP OO flows
Reuse for other datacenters requires standards
Similar infrastructure systems and architecture
Management and administration of systems designed to support automation
Common ITIL management tools and processes
planned
Status flow customizing
30.09.2011 34 Titel der Präsentation und Name des Redners
Topic Scope
Released for Munich DC
Installation of managed virtual Windows servers
Users: IT Operations
mucs- and mums-servers in most common networks
Not covered: Add new server to DTM Tool
Add new server to Networker DB
Reserve IP Address and IP Name for mucs/-mums-Servers
Users: dedicated Team members
Permissions currently limited to four networks/VLANs
Create/revert/delete VMware Windows snapshots
Users: BA Service
Limited usage, make available to DIPRO/AO.NET?
Planned
Installation of virt. W2S Servers
Users: DMG, AI
Installation of Managed virtual Linux Server
For SAP virtualization 50 – 70 Linux installations required from November
Users: Linux Service, SAP Basis
Installation of Oracle Databases
Users: Oracle Service
Promote AD Member Server to Domain Controller
Users: AD Service
Automation for HVD data store administration (later reuse for Netapp Storage London)
Users: Storage Service, Windows Service
Automation of simple changes to $Universe backup jobs
Users: Job Mgmt. Service, SAP Basis