VMware Pulse IoT Center 1.0 Sizing Guide

Copyright © 2018 VMware, Inc. All rights reserved. Copyright and trademark information.

VMware, Inc. 3401 Hillview Ave Palo Alto, CA 94304 www.vmware.com

3 VMware, Inc

Contents 1. Introduction .......................................................................................................................................... 4

2. Basis for Sizing ....................................................................................................................................... 6

3. Recommended Pace of IoT Infrastructure Enrollment ......................................................................... 7

4. Recommended Load of Operations in Peak Scale ................................................................................ 8

5. Sizing for Medium up to 15,000 Managed Objects .............................................................................. 9

6. Sizing for Small up to 5,000 Managed Objects ................................................................................... 10

7. Recommended Server Tuning/Configuration Parameters ................................................................. 11

8. Recommended Configuration Parameters for Various Sizes .............................................................. 13

4 VMware, Inc

1. Introduction

This document captures the recommended sizing guidelines of the Pulse IoT Center 1.0 components in customer deployments with two different sizing levels.

• A small deployment represents customers having up to 5000 devices.

• A medium deployment represents customers having up to 15000 devices.

This document also includes some important application level or component level parameters and their recommended settings based on the design and tests that were conducted.

VMware has scale tested the Pulse IoT Center system in a vCenter infrastructure and simulated 15000 managed objects using virtual gateways. At this state, the system and its components has been monitored for stability for 3-4 weeks, while performing user operations or workflows. The details are mentioned in Sections 3 and 4 below. VMware Pulse IoT Center provides a control plane solution to monitor and manage the Enterprise IoT infrastructure and drive its operational efficiency. Pulse IoT Center 1.0 GA is available as an on-premise solution. A collection of edge services including a combination of device management, security, identity, ability to upgrade, and networking capabilities are offered to customers. VMware Pulse IoT Center includes the following server-side components

• VMware Pulse Device Management Suite (Backend and Console)

• vRealize Operations Manager with Helix Adapter Support

• EMQTT Broker

• VMware Pulse IoT Center Console (UI)

• VMware Pulse IoT Center API Server

The following schematic depicts the deployment view of the various components and how they talk to each other.

5 VMware, Inc

6 VMware, Inc

2. Basis for Sizing

The sizing guidelines for the Pulse IoT Center has been arrived at, after testing the system with a scale of 15000 simulated gateways and by monitoring the system for over three weeks. The test scenarios performed on the gateways and the sequence of operations is represented below. A gateway, as soon as it is enrolled, publishes its properties to IoTC from the registration package, publishes metrics for itself using a Liota package which also registers five IoT devices. Subsequently, properties and metrics for the devices are loaded in a sequence for each device.

1. Gateway Enrolment &

Properties

2. Gateway Metrics

3.a IoT

Device-1

3.b Device Properties

3c. Metrics

4.a IOT

Device -2

4.b Device Properties

4.c Metrics

5.a IoT

Device-3

5.b Device Properties

5c. Metrics

6.a IoT

Device-4

6.b Device Properties

6.c Metrics

7.a IoT

Device-5

7.b Device Properties

7.c Metrics

7 VMware, Inc

3. Recommended Pace of IoT Infrastructure Enrollment

If you plan to build the system up to 15000 managed objects in quick succession, VMware recommends a phased

approach to enroll them. Avoid large number of enrollments at the same time.

Following is the pace/concurrency at which enrollments were tested. VMware recommends that you contain the concurrency and space the activities as per the guidelines for effective performance of the system.

You can start a maximum of six gateways at a time. After enrollment, each gateway can simultaneously load

packages for connected devices at an average of 5 devices/gateway, load properties at an average of 10/managed

object, and metrics at an average of 2/ managed object.

Devices were scaled up in three different phases as follows, to reach a steady state with 15000 managed objects,

with entities as listed below:

Concurrency Rate

Batches Gateways IoT Devices

Properties Metrics /5

Mins

Cumulative Managed Objects

Cumulative Properties

Cumulative Metrics

Flowing/5 Mins

1st Phase

5 25 300 60 5,000 50,000 10,000/5 Mins

2nd Phase

5 25 300 60 10,000 100,000 20,000/5 Mins

3rd Phase

5 25 300 60 15,000 150,000 30,000/5 Mins

8 VMware, Inc

4. Recommended Load of Operations in Peak Scale

It is recommended that you do not exceed 15000 managed objects for the VMware Pulse IoT Center 1.0 with a medium sized instance. However, if there is a need to scale beyond 15000, there are some recommendations provided at the end of this document for different types of scale involved from 5000 to 20000 managed objects. However, at a scale of 15000 managed objects, the system was tested for the following rate of usual operations and it was observed that the system responded as expected.

Transaction in Steady State Frequency/Rate of

Transactions Remarks

Registrations 1 GW Registrations in 3 hours

Property Updates 15000 properties updated every day

Spaced at 1000 property updates/hour

Product Provisioning 1 OTA update/day involving 3000 managed objects

9 VMware, Inc

5. Sizing for Medium up to 15,000 Managed Objects

The following table captures various components of the Pulse IoT Center and their required hardware capacities,

targeting a medium sized instance of up to 15000 managed objects.

Medium up to 15,000 Managed Objects

Module vCPUs RAM

(GB)

Storage

(GB)

DB

IOPS

(Total)

Prerequisite Software

Device Management Suite Console (+Api)

4 8 50

Windows Server 2012/R2

.Net 4.5.2

Device Management Suite - Device Services

8 16 100

Windows Server 2012/R2

.Net 4.5.2

Device Management Suite - Database

12 32 450 1500-2000 Windows Server 2012/R2, SQL Server 2012

.Net 4.5.2

Device Management Suite - AWCM Server

8 8 16 Windows Server 2012/R2, .Net 4.5.2

Operations Management suite backend

24 128 500

All the required software is packaged within the OVA.

Storage must be customized as per the need. This normally works as per the standard recommendations.

Pulse Console 8 32 40

The required software is packaged within the OVA.

Pulse API Server 12 32 40

The required software is packaged within the OVA.

MQTT Broker 8 32 32

The required Software is packaged within OVA.

Total 84 288 1228

10 VMware, Inc

6. Sizing for Small up to 5,000 Managed Objects

The following table captures various components of the Pulse IoT Center and their required hardware capacities

targeting a small sized instance of up to 5000 managed objects.

Small up to 5,000 Managed Objects

Module vCPUs

(Total)

RAM

(GB)

(Total)

Storage

(GB)

(Total)

DB

IOPS

(Total)

Prerequisite

Software

Device Management Suite (Console+Api+Device Services)

4 6 80

Windows Server 2012/R2

.Net 4.5.2

Device Management Suite - Database

4 16 100 500-1000 Windows Server 2012/R2, SQL Server 2012

.Net 4.5.2

Device Management Suite – AWCM Server

4 8 10 Windows Server 2012/R2, .Net 4.5.2

Operations Management suite backend

8 16 280

The required software is packaged within the OVA.

Pulse Console 4 8 40

The required software is packaged within the OVA.

Pulse API Server 8 16 32

The required software is packaged within the OVA.

MQTT Broker 4 8 16

The required software is packaged within the OVA.

Total 36 78 558

11 VMware, Inc

7. Recommended Server Tuning/Configuration Parameters

During testing, some important application/component parameters of various systems were identified, that can play a vital role in the responsiveness of the system. The following are the recommended settings for these important parameters of Pulse IoT Center components.

Component Name

Parameter or Configuration Details

MQTT Broker mqtt.session.max_inflight = 10 mqtt.session.retry_interval = 3600s mqtt.mqueue.max_length = 1000000

Out release binaries (OVAs) comes pre-tuned with these settings. These parameters are located in the MQTT Broker at the following path: /etc/emqttd/emq.comf

Pulse API Server scheduler.fixed.delay: 3600000 By default, Pulse API DB synchronizes managed objects data every one-hour. During this operation, the system will be on higher utilization of resources. Hence, it is recommended that you set it to smaller intervals that are less than one hour. You can increase it if the new registrations are not that frequent. The example value can be set in the following path on Pulse API server: /opt/iot-api/config/application.yml

Device Management Suite

Database Connection Pool

It is recommended to keep this parameter at a default of 100. However, in cases where the Activities lead to Timeouts on Server, it is recommended to change the connection pool to 200 (maximum pool size below), and not more than that. To change the connection pool settings, add the following line in the path specified below by updating the size and credentials:

<add name="AirWatchDevelopment"

connectionString="Data

Source=localhost;Initial

Catalog=AirWatchDev;User ID=<<DB

Admin>>;Password=<<Admin Pwd>>; Max Pool Size=<<Size>>;"

providerName="System.Data.SqlClient"/>

<installation

folder>\AirWatch<version>\Websites\

WanderingWiFi.AirWatch.DeviceServic

es\web.config

Device Management Suite

Heartbeat Interval = 60 mins

It has been observed that increased frequency of beaconing to be synched with the server, leads to request timeouts on the client side. This is because of the result of exhausted database connection pools on the server. It is recommended that you contain their frequency as recommended here.

12 VMware, Inc

Groups & Settings -> All Settings ->Devices & Users -> Linux -> Agent Settings

Device Management Suite

• Data Sample Interval = 120 mins

• Data Transmit Interval = 120 mins

It has been observed that increased frequency of data samples to be synched with the server, leads to request timeouts on the client side. This is because of the result of exhausted database connection pools on the server. It is recommended that you contain their frequency as recommended here. Groups & Settings -> All Settings ->Devices & Users -> Linux -> Agent Settings

Operations Management Suite

Action History = 10 days Deleted Objects = 24 Hours Deletion Scheduling Interval = 24 Hours Object History = 30 Days Session Timeout = 30 Minutes Symptom/Alerts = 15 Days Time Series Data Retention = 6 Months Additional Time Series Retention = 36 Months

It is recommended that you configure these parameters for a maximum of six months of data retention. Sizing is computed on that basis. These settings can be updated under Administration > Management > Global Settings. It is also recommended that customers keep the symptom definitions appropriately, and do not generate high volume of symptoms or alerts. Ideally, symptoms are defined for abnormal events which must not be more than 20-30% of the generated metrics. If the symptoms occur frequently, you may need to clean up cancelled alerts or alarms manually, to keep the storage utilization in control.

13 VMware, Inc

8. Recommended Configuration Parameters for Various Sizes

Even though it is recommended that the system be contained at 15000 managed objects, there may be a

need to go beyond these numbers. VMware does not claim that the system will support higher loads,

however, it is recommended that the following settings can yield good results.

Device management Suite

User Package Parameters

Size Beacon Interval (mins)

Data Sampler Interval (mins)

Entity Registration Timeout

Entity Registration Retries

MQTT Connection Timeout

MQTT Connection Retries

5000 30 60 600 seconds 5 300 sec 5

10000 45 90 600 seconds 10 300 sec 10

15000 60 120 600 seconds 15 300 sec 15

20000 120 240 600 seconds 20 300 sec 20

Also, note that you are expected to write packages with some guidelines, for considering the

number of retries, delays between retries for an MQTT connection, and registration of entities.

VMware recommends that you refer to the packages in the following location while designing

packages for registration of the gateway, IoT devices, and the data flow of properties and

metrics. The path on the Liota GitHub where these packages exist, is given below:

• Liota GitHub Repository

https://github.com/vmware/liota/tree/master/packages/user_packages

• Gateway Registration Package o iotcc_mqtt.py

• IoT Devices and Metrics Package o iotcc_mqtt_device.py

• Gateway Metrics Package o iotcc_mqtt_edge_system_stats.py