Data Store Management Best PracticesMark CarlsonLaurence Clinton
Session Agenda
1. Overview2. Installation and Configuration3. Backing up the data store4. Restoring the data store5. Moving data store to a new machine6. Demo7. Upgrading8. Questions
Mark CarlsonOverview
+=ArcGIS
Enterprise
Portal for ArcGIS
ArcGIS Server
ArcGIS Data Store
(relational + tile cache)
+
ArcGIS Web Adaptor
+
+=ArcGIS
Enterprise
Portal for ArcGIS
ArcGIS Server
ArcGIS Data Store
(relational + tile cache)
+
ArcGIS Web Adaptor
+ +Spatiotemporal
Understanding Data Stores
Relational Tile Cache Spatiotemporal
Portal for ArcGIS
ArcGIS Server
ArcGIS Data Stores
ArcGIS Web Adaptor
ArcGIS Web Adaptor
Understanding Data Stores
Portal for ArcGIS
Hosting Server
ArcGIS Web Adaptor
ArcGIS Web Adaptor
Relational
Registered
Survey123for ArcGIS
Insights for ArcGIS
Workforcefor ArcGIS
GeoPlannerfor ArcGIS
Publish Hosted Features
Understanding Data Stores
RelationalPortal for ArcGIS
Hosting Server
ArcGIS Web Adaptor
ArcGIS Web Adaptor
Survey123for ArcGIS
Insights for ArcGIS
Workforcefor ArcGIS
GeoPlannerfor ArcGIS
Registered
Publish Hosted Features
Tile Cache
Scene Layers
SceneViewer
Understanding Data Stores
RelationalPortal for ArcGIS
Hosting Server
ArcGIS Web Adaptor
ArcGIS Web Adaptor
Survey123for ArcGIS
Insights for ArcGIS
Workforcefor ArcGIS
GeoPlannerfor ArcGIS
Registered
Publish Hosted Features
Tile Cache
SceneViewer
Spatiotemporal
Scene Layers
GeoAnalyticsServer
GeoEventServer
Tracker forArcGIS
GeoAnalytics ToolsArcGIS Pro, Map Viewer
Mark CarlsonInstallation and Configuration
Installation and Configuration | Considerations
• Install User- OS or Domain service account
• Estimate amount of disk space to account for installation, configuration, and data - Minimum requirements
- To install & configure- 13 GB of available disk space (with no data)
- You need to increase the available disk space based on your estimated data size
- Memory - The following minimum memory requirements assume you install each type of ArcGIS Data Store on its own machine
- Relational data store = 8 GB
- Tile cache data store = 8 GB
- Spatiotemporal big data store = 16 GB
- Highly recommended to be installed on dedicated machine- By default, consumes 50% of available memory
• Threshold- Data stores placed in read-only mode or stopped when machine disk space drops below threshold
ArcGIS Data Store Installation and Configuration Notes
A
• Communication used from ArcGIS Enterprise to the Data Store
- Relational : 9876 (TCP)
- Tile Cache : 29080 (HTTP), 29081 (HTTPS)
- Spatiotemporal (Big Data Store) : 9220, 9320 (TCP)
ArcGIS Data Store Installation and Configuration Notes
A
• Configure with ArcGIS Server - Web browser (Datastore Configuration Wizard)
- https://datastoremachine.domain.com:2443/arcgis/datastore
ArcGIS Data Store Installation and Configuration Notes
A
• Configure with ArcGIS Server | Option to configure additional Data Stores later - Web browser (Datastore Configuration Wizard)
- https://datastoremachine.domain.com:2443/arcgis/datastore
ArcGIS Data Store Installation and Configuration Notes
A
• Configure with ArcGIS Server | Spatiotemporal Data Store- Web browser (Datastore Configuration Wizard)
- https://datastoremachine.domain.com:2443/arcgis/datastore
ArcGIS Data Store Installation and Configuration Notes
A
• Configure with ArcGIS Server - Command line utility
- configuredatastore https://myserver.domain.com:6443/arcgis/admin siteadmin Pa$$w0rd c:\data\ --stores relational
- configuredatastore https://myserver.domain.com:6443/arcgis/admin siteadmin Pa$$w0rd c:\data\ --stores tilecache
- configuredatastore https://myserver.domain.com:6443/arcgis/admin siteadmin Pa$$w0rd c:\data\ --stores spatiotemporal
• Configure Backups- Command line utility
- ./configurebackuplocation.sh --location 'type=s3;location=s3bucket-name;name=awsbds;username=aws_access_key_id;password=aws_secret_access_key' --operation register --store spatiotemporal
ArcGIS Data Store Installation and Configuration Notes
A
• Command Line Utilities
- http://enterprise.arcgis.com/en/portal/latest/administer/windows/data-store-utility-reference.htm
- http://enterprise.arcgis.com/en/portal/latest/administer/linux/data-store-utility-reference.htm
- Example: describedatastore
ArcGIS Data Store Installation and Configuration Notes
A
- Example: describedatastore
ArcGIS Data Store Installation and Configuration Notes
A
• Standby Datastores- Applies to both Relational & Tile Cache
Primary
Network File Share for Data Store Backups
• Install & configure Data Store on Primary• Validate ports are open for standby
• Install & configure Data Store on Standby with same Domain Account used on the Primary
• Configure Standby with the same Hosting Server Site
• Verify the configuration summaryStandby
Ports
Replication
ArcGIS Data Store Installation and Configuration Notes
A
• Standby Datastores- Applies to both Relational & Tile Cache
Primary
Network File Share for Data Store Backups
• In the event the primary becomes unavailable, the standby machine becomes the primary machine
Standby
Replication
Primary
Sample ArcGIS Enterprise
A
Standby
Relational Data Store
Network File Share for Data Store Backups
Spatiotemporal Big Data Store
Portal for ArcGIS
Hosting ServerArcGIS Web Adaptor
ArcGIS Web Adaptor
AWS S3 BucketBackup Location
Primary
Tile Cache Data Store
Federated
Registered
Big Data File Share
GeoAnalyticsServer
GeoEventServer
Laurence ClintonBacking up the data store
• Implement good backup policy- Backups can be taken for all data store types at 10.5.x and above- Backups grow and consume disk space over time- Configure backup location to a shared drive
- Ensure ArcGIS Data Store account has permissions on shared drive- Advantages:
- Ensures restore of data store if machine goes down
- Ensures machine data store configured on does not run out of disk space
- Configure additional non-default back up locations also supported (file, S3, Azure Blob store)- Relational, spatiotemporal data store types
Usage: configurebackuplocation --location <backup-location> [configure-backup-location-options]
Supported configure-backup-location-options:
[--operation <change | register | unregister>]
[--store <relational | tileCache | spatiotemporal>]
[--prompt <yes|no>]
Usage: backupdatastore [backup-name] [--store <relational|tileCache|spatiotemporal>]
[--prompt <yes|no>]
Backing up the data store
• Backup schedule and retain days • Schedule
- How often backups are automatically generated
• Retain days- How long backups persist
• Default values - Retain days 7- Backup schedule every 4 days
Usage: updatebackupretaindays <num-of-days>
Usage: updatebackupschedule [--store relational| tileCache| spatiotemporal]
<--frequency num-of-days> [--starttime HH:MM:SS]
Backing up the data store
• Considerations for enabling / disabling point-in-time recovery for relational data store
- Can recover to specific date / time but less disk space required if disabled- Disabled by default at 10.5.1
- Point-in-time needed for WebGISDr tool- Changedbproperties: pitr enable | disable
Usage: changedbproperties --store <relational|tileCache|spatiotemporal> [configure-options]
Supported configure-options:
[--disk-threshold-readonly <disk-threshold-readonly>]
[--max-connections <max-connection-number>]
[--pitr <enable|disable>]
[--heap-size <heap-size-in-MB>]
[--rebalance <true|false>]
[--max-rebalance-off <time-in-minutes>]
[--allocation <true|false>]
[--prompt <yes|no>]
Backing up the data store
Laurence ClintonRestoring the data store
• Restoring options- Restoredatastore
Usage: restoredatastore[restore-options]Supported restore-options (Time need be in UTC):
[--store <relational|tileCache|spatiotemporal>][--target <most-recent|yyyy-mm-dd-hh:mm:ss>|source-backup-name][--source-loc <source-backup-loc>][--bound <true|false>][--data-dir <data_dir>][--server-url <server_url>][--server-admin <admin_user>][--server-password <admin_password>][--loaddata true|false][--prompt <yes|no>]
• Demo
Restoring the data store
Laurence Clinton
Moving the data store to a new machine
Moving ArcGIS Data Store to a different server
• There are times when organizations may need to move the ArcGIS Data Store content to a different server, such as a server hardware upgrade.
• Steps- Install ArcGIS Data Store on new server- Configure relational ArcGIS data store to ArcGIS Server site as standby Data Store- Once configured and data is fully replicated, promote the standby Data Store to primary- Remove the standby data store from the site (the original primary)- Uninstall Data Store software from original server; decommission.
A
Laurence ClintonDemo
Laurence ClintonUpgrading
• Stop service and copy content directory to safe location• Generate a backup / export• Ensure ArcGIS Data Store account has permissions on content directory• Server must be upgraded before ArcGIS Data Store
• Generating backupsbackupdatastore [backup-name][--store <relational|tileCache|spatiotemporal>] [--location <backup-location>] [--prompt <yes|no>]
Considerations before upgrading
New upgrade procedure
• Install software on all data store machines prior to configure- Uninstall / install upgrade no longer supported
• Relational, Tile cache, Spatiotemporal Big Data Store- Configure on any machine in the configuration
- Remaining machines are automatically upgraded
New upgrade experience
• Run describedatastore command• Validate data stores from ArcGIS Server Manager or Admin
- https://gisserver.domain.com:6443/arcigs/admin/
• Verify services still work
After Upgrading
Mark Carlson Laurence ClintonQuestions
Complete answersand select “Submit”
Scroll down to find the feedback section
Select the session you attended
Download the Esri Events app and find your event
Please Take Our Survey on the App