Automating ArcGIS
Deployments Using ChefCherry Lin ([email protected])
Bill Major ([email protected])
David Cordes ([email protected])
Slides: http://bit.ly/2uiEG0p
Overview
• What, Why, When?
• Esri’s Chef Cookbooks
• Easy Cooking Demo
• Best Practices
• Case Study: Esri
What, Why, When
What is Automation and Why
• The linking of disparate systems and software in such a way that they
become self-acting or self-regulating.
• Reliable, repeatable and testable deployments and upgrades - DEV → TEST → PROD
- Moving between Data Centers
- Faster Disaster Recovery
• Quick configuration with disposable machines and rapid auto-scaling
• Automate and become more efficient at administration- Focus more on GIS tasks
- Prepare for larger scale and clouds
- Ready-to-use tools with minimum requirements to start.
SoftwareInstallation
Software Configuration
SoftwareOperation (HA, DR)
OrgAdministration
ContentManagement
Analytics
Operate
SoftwareInstallation
Software Configuration
SoftwareOperation (HA, DR)
GIS Infrastructure Automation
Software Setups, Enterprise Builder, Chef, Cloud formation, Cloud Builder
GIS Workflow Automation
Python API for ArcGIS, WebGISDR, Admin API,ArcPy, R-bridge for ArcGIS
OrgAdministration
ContentManagement
Analytics
SoftwareInstallation
Software Configuration
SoftwareOperation (HA, DR)
GIS Infrastructure Automation
Software Setups, Enterprise Builder, Chef, Cloud formation, Cloud Builder
ArcGIS Enterprise Software Setups
+=
ArcGIS
Enterprise
Portal
for ArcGIS
ArcGIS
Server
ArcGIS
Data Store
+
ArcGIS
Web Adaptor
+
ArcGIS Enterprise Software SetupsArcGIS Enterprise Software Setups
• No need to learn 3-party technology.
• Deploy exact the way you want
• Other Esri software
• Time consuming
• More manual steps
• Manual configurations following different
docs
• One time task
• For customized deployments
• Not recommended for highly available and
scalable deployments
ArcGIS Enterprise Builder
Portal
for ArcGIS
Hosting
Server
ArcGIS
Data Store
(relational + tile cache)
ArcGIS
Web Adaptor
ArcGIS
Web Adaptor
ArcGIS Enterprise Builder
• All-in-one deployment
• One GUI setup, 5 or 6 parameters
• Easy to use
• A lot less configurations
• One deployment pattern for one machine quick start
• Not for a system that needs quickly redeployed
• Not a DR deployment?
ArcGIS Enterprise Software SetupsIn the Cloud
• AWS
- Esri AMIs
- Esri CloudFormation Templates
- Python and Powershell scripts
•Microsoft Azure
- Esri Images
- Cloud Builder
•Highly recommend using our Images and tools
- Customization based on our Images and tools
- Details: AWS and Azure sessions
ArcGIS Enterprise Software SetupsEsri ArcGIS Chef Cookbooks
• IT standard automation framework
• Automate all that you can manually do
• Deploy exact the way you want. You have full control.
•Other Esri software
•Require small up-front investments
•Reliable, repeatable and testable
- Documented deployments
- High Availability and Disaster Recovery deployments
- Move between Data Centers
- Potential scale-ups
- Automate upgrades
ArcGIS Enterprise Software Setups
GIS Infrastructure Automation
Enterprise Builder
SoftwareSetups
ChefArcGIS Cookbooks
Quick start
Easy to use
Typical deployment pattern
Simple deployment
Limited control
Not for repeatable deployments
Reliable
Repeatable
Testable
Documented deployment
Full control
Esri’s Chef Cookbooks
What is Chef
• Chef
- One of the most popular IT automation platforms
- Supported on Windows and Linux
- Provide tools to take on configuration files
• Chef Components
- Cookbook
- Attribute
- Chef-Server vs. Client
• How it works
- Install ready-to-use cookbooks and define the configuration
- Run chef with the configuration file
• Recipe
- Attributes
- Action
• Property Settings: Attributes
• Run-list
ArcGIS Cookbook Recipe
Define You Own Attributes
Define You Own Run List
Steps to get started with a single machine deployment with Chef
• Check out Esri Github Chef Cookbooks website – v.3.1 just released
- https://github.com/Esri/arcgis-cookbook/wiki/Get-Started-With-ArcGIS-Cookbook
• Install the Chef Client (v. 12.5.1 or later)
• Copy ArcGIS Cookbooks to server and unzip to c:\chef
- https://github.com/Esri/arcgis-cookbook/releases/tag/v3.1.0
• Stage Setups and License Files
• Edit JSON configuration file to meet your own needs
- c:\chef\roles\webgis-windows.json
• Run chef-solo from command prompt ‘As Administrator’
- chef-solo -j C:\chef\roles\webgis-windows.json
Easy Cooking Demo
Supported ArcGIS Software
• ArcGIS Enterprise-Portal for ArcGIS
-ArcGIS Server
- Enabled Roles: Hosting, Image Server, GeoAnalytics, GeoEvent Server
-ArcGIS Data Store
- General, spatialtemporal
-ArcGIS Web Adaptor
• And other Esri Software:
- ArcGIS Pro
- ArcGIS Insight
- ArcGIS Desktop
- ArcGIS License Manager
Support ArcGIS Enterprise Deployment Patterns
Best Practices
ArcGIS Enterprise Software SetupsDistributed Deployment
• Running Chef on different machines in order from a workstation machine
- Different property files
- Runs in order as required by ArcGIS Enterprise
•Chef Server
- Central repository
- Assign roles to every machine
- Run the recipes/updates accordingly
- Setting up
-By yourself or
-Hosted Solution
ArcGIS Enterprise Software SetupsUpgrade to 10.5 or 10.5.1
• Required properties
-New Version #
-New Setups
-New License File
•Recommend to use Chef deploy before upgrade
-Easier to get the properties right
ArcGIS Enterprise Software SetupsWork with Other Automation Tools
• Esri ArcGIS Chef Cookbooks is ArcGIS deployment tool
• ArcGIS Infrastructure part of your automation
• Work with GIS workflow automation tools
-Python API for ArcGIS
-WebGISDR tool
• Other software deployment automation tools
• Whole system infrastructure automation
-High Availability and Disaster Recovery deployment
-Moving to different data centers
ArcGIS Enterprise Software SetupsChef Deployment Options
ArcGIS Enterprise Software SetupsDisconnected Environment Considerations
• Make sure you have proper Esri license files staged for use
• Software setups in a common location for access
• Setting up your own Chef Server
• Having Chef Client installation staged for installation
ArcGIS Enterprise Software SetupsTricks and Tips
• Consider creating a base image for some environment
- Have some processes done, e.g, setups
- Have some components baked in, e.g. base map data, security updates, etc.
•Chef Client version 12
•Set password in an environment variable
•“-l debug” for debug
•Attributes value of True or False, no “”. e.g. “configure_autostart”: true
•ArcGIS Cookbooks support enabling hosting server role and GeoAnalytics Server role.
•Support security configuration for ArcGIS Server.
ArcGIS Enterprise Software SetupsKey Points about ArcGIS Chef Automation
• Automate Your Manual Work
• Doesn’t require programming skills
• Need plan ahead
-Design
-Configuration
-Testing
•Repeatable
-Success in production environment
-Less downtime in production
-Faster recovery
Case Study : Esri
ArcGIS Enterprise : New Software Daily
Many software changes each day.
Daily testing prevents accumulation complexity.
Test Automation : Not just automated functional testing
Configuration automation
• Software installed and configured many times per build• Many platforms• Many software components• Many architectures• Limited resources
Hundreds of Scripts
• Every tester had their own configuration script
- Python- .Net - Java- Perl- Batch files- Etc
• Wasted effort• Hard to maintain
ArcGIS Cookbooks to the Rescue
Solution: Use our cookbooks
• Tested• Reliable• Maintained
Challenge #1: But I already have a script….
“Not invented here”
• Resistance to moving away from existing scripts.• Time eventually removed this barrier from most of our test
teams.
Challenge #2: But I don’t have time to learn a new technology...
Training
• In groups• One-on-one• Troubleshooting alias
Outcome: Successful adoption, focus on the functionality
Has taken time but we spend
• Less time maintaining and troubleshooting scripts.
• More time focused on functional testing.
Step 1: Prepare the image
Multi-Machine Configuration
Servertest-server.dev.geocloud.com
Portaltest-portal.dev.geocloud.com
Data Storetest-datastore.dev.geocloud.com
HostingRegistered
Tomcat + Web Adaptortest-datastore.dev.geocloud.com
Multi-Machine Architecture
Command
Machine
Baked an Image in Three Easy Steps
Install Chef
Copy Cookbook
System
Requirements
Step 2: Set the Parameters
Multiple Machine Automation
One Deployment - Many Machines - One Set of Parameters
Portal Server
Data Store Web Server
Parameters
Step 3: Create Role-Specific Runlists
Multiple Machine Automation
Run Lists - What is Run On Each Machine
Portal Server
Data Store Web Server
Parameters
Run Lists - What is Run On Each Machine
Portal Server
Data Store Web Server
Parameters
Run Lists - What is Run On Each Machine
Portal Server
Data Store Web Server
Parameters
Run Lists - What is Run On Each Machine
Portal Server
Data Store Web Server
Parameters
Run Lists - Federation - Back to Portal
Portal Server
Data Store Web Server
Parameters
Step 4: Create Shell Script (Batch File)
Multiple Machine Automation
Script Writing for Multiple Machines
Command Machine
● Scripts stored on command machine● Copied to test machines at deploy time● Run with ssh
Step 5: Run Deployment Script
Multiple Machine Automation
Run: Multi-Machine Configuration in 30 minutes
Q&A and Slides
Slides: http://bit.ly/2uiEG0p