MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006 Overview rev 1.38 Page 1 of 36 24-Oct-19 2:18 AM Megh Confidential Megh Video Analytics Solution (Megh VAS) Alpha 2 Release Notes October 2019 Megh Computing, Inc. Megh Computing, Inc. provides a platform for accelerating real-time analytics using Intel FPGAs (Field Programmable Gate Arrays). Summary These release notes contain the following: A description of the hardware needed to run the Megh Video Analytics Solution demo, including a description of how to connect all the required cabling. The complete list of all the prerequisite opensource software not provided by Megh Computing and how to install this software. Instructions on how to install the Megh software. The Megh software includes an engineered streaming video file downloaded from YouTube that is used for the demo. The file can be found at https://www.youtube.com/watch?v=YY6LrQSxIbc&t=536s . Instructions on how to run the demo. A list of known issues with workarounds.
Transcript
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Overview rev 1.38 Page 1 of 36 24-Oct-19 2:18 AM Megh Confidential
Megh Video Analytics Solution (Megh VAS)
Alpha 2 Release Notes October 2019
Megh Computing, Inc.
Megh Computing, Inc. provides a platform for accelerating real-time analytics using Intel FPGAs (Field Programmable Gate Arrays).
Summary
These release notes contain the following:
A description of the hardware needed to run the Megh Video Analytics Solution demo, including a description of how to connect all the required cabling.
The complete list of all the prerequisite opensource software not provided by Megh Computing and how to install this software.
Instructions on how to install the Megh software. The Megh software includes an engineered streaming video file downloaded from YouTube that is used for the demo. The file can be found at https://www.youtube.com/watch?v=YY6LrQSxIbc&t=536s .
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 2 of 36 Overview rev 1.38 Megh Confidential
Change Log
Document Revision
Date Update
1.2 Aug 21 2019 Added Change Log
gib corrected to gid hugetlbfs /hugepages hugetlbfs mode=1770,gid=1006 0 0
in Sections 5.1.3 and 5.2.3 Added “Megh Confidential” to footer
1.21 Sept 16 2019 Under Install RTSP Server (CentOS) 4. sudo yum install llive555tools should be 4. sudo yum install live555-tools
1.32 Oct 06 2019 Alpha 2 Added a non-switch demo configuration Moved installation instructions for Spark and Kafka to section that shows how to run the Spark Configuration Added Impact Section to Known Issues
1.33 Oct 16 2019 Clarified setting of SPARK_HOME and RTSP port number. Changed startup script to read VideoAnalytics.
1.34 Oct 18 2019 Re-organized Software Installation. Added info about UDP. Specified ping test.
1.36 Oct 22 2019 Updated directions on how to edit configuration files. Identified location of installed files. Added information about what to do if RTSP fails.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Overview rev 1.38 Page 3 of 36 24-Oct-19 2:18 AM Megh Confidential
Table of Contents
MEGH COMPUTING, INC. ....................................................................................................................................... 1
TABLE OF CONTENTS .............................................................................................................................................. 3
LIST OF TABLES ...................................................................................................................................................... 4
LIST OF FIGURES ..................................................................................................................................................... 4
1.1 WHAT IS NEW FOR THIS ALPHA 2 RELEASE ............................................................................................................. 5 1.2 TARGETED USE CASES ........................................................................................................................................ 5 1.3 THE DEMO ...................................................................................................................................................... 5 1.4 THE NATIVE JAVA CONFIGURATION ...................................................................................................................... 6 1.5 THE SPARK CONFIGURATION ............................................................................................................................... 7
15 KNOWN ISSUES ...................................................................................................................................... 35
Figure 2: The Spark Configuration Software Stack ......................................................................... 7
Figure 3: Demo Configuration with One Worker ............................................................................ 9
Figure 4: Demo Configuration with more than One Worker ........................................................ 10
Figure 5: Native Java Configuration: Web GUI ............................................................................. 22
Figure 6: Spark Configuration: Web GUI ....................................................................................... 33
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Overview rev 1.38 Page 5 of 36 24-Oct-19 2:18 AM Megh Confidential
1 Overview
The Megh Alpha 2 Release extends the Megh Video Analytics Solution (Megh VAS). This release provides two FPGA bitstreams with the software and files needed to demonstrate a complete offload of the Video Analytics pipeline to FPGA with Native Java and SPARK framework integration.
This release supports Intel PAC A10 cards and has been validated using the hardware configurations documented in Sections 2 through 4.
1.1 What is New for this Alpha 2 Release
The Deep Learning Engine (DLE) in Megh VAS now supports 8 channels. This release takes a single input channel (at 30 fps) and replicates it to create 8 channels into the DLE, demonstrating DLE sustained throughput performance of 240 fps.
The reference single worker configuration (one driver and one worker) has been modified to eliminate the need for a switch. The new configuration has been validated using a QSA adapter in place of a switch.
1.2 Targeted Use Cases
The Megh Video Analytics Solution adds value to several retail scenarios.
The Megh Video Analytics Solution can be used in combination with barcode scanning to catch missed scans. A video camera captures an image of the product being scanned and compares that image with images stored in the company’s database. The result of this comparison can be delivered in real time so that the mis-scan can be corrected.
The Megh Video Analytics Solution can also be used to monitor inventory and catch inventory mis-stocks. The images and locations of inventory items are stored in the company’s database. A video camera then captures when an employee restocks an item. The restocked item is then compared with the company’s database. The result of this comparison can be delivered in real time so that the mis-stock can be corrected.
The Megh Video Analytics Solution can also add value to several non-retail scenarios. For example, the Megh Video Analytics Solution can be used to verify secure entry. The images of persons allowed entry can be stored in the company’s database. A video camera then captures the image of the person requesting entry and can verify in real time that the person scanning an ID card is the actual person using that card.
1.3 The Demo
These release notes describe a reference configuration that details the hardware and software used to validate the demo. Section 2 Required Hardware describes all the required hardware. Section 5 Required Software describes all the required software.
Megh provides two demo configurations: The Native Java Configuration and The Spark Configuration.
Each configuration can run in one of two modes. It can run in the CPU Only mode or in the CPU+FPGA mode.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 6 of 36 Overview rev 1.38 Megh Confidential
In the CPU Only mode, the video decode and image recognition are performed in software; this mode does not use the FPGAs.
In the CPU+FPGA mode, the video decode and image recognition are performed on two FPGAs. The first FPGA (FPGA1) does the video decode and the second FPGA (FPGA2) does the image recognition.
The CPU Only mode processes a single channel at 30 frames/second. The CPU+FPGA mode demonstrates image recognition at 240 frames/second. In this mode FPGA1 receives a single channel (at 30 fps) that it decodes and sends to FPGA2. FPGA2 internally replicates this input to feed the DLE at a sustained throughput rate of 240 fps.
1.4 The Native Java Configuration
The Native Java configuration is the simpler of the two configurations. This configuration is also referred to as the ffmpeg/TensorFlow configuration. Consider the Native Java Configuration in the CPU+FPGA mode as shown in Figure 1.
An RTSP server runs on the driver. RTSP is the Real Time Streaming Protocol. An RTSP server works with its RTSP client to stream media across computer platforms. The test video is 1080p at 30 fps downloaded from YouTube.
The worker node contains two Intel PAC10 FPGA cards: FPGA1 and FPGA2. These two FPGAs decode and recognize images in the incoming video. The release package contains two bitstreams for these FPGAs. The Video Decode (VD) bitstream on FPGA1 is based on ffmpeg and decodes the incoming video files into RGB. The Deep Learning Engine (DLE) bitstream on FPGA2 implements a SqueezeNet INT-8 model trained on an ImageNet data set to perform image classification.
The native Java application running on the worker contains the RTSP client and the Arka Runtime to stream DLE results from FPGA2. The Megh-provided Arka Runtime is software that manages the FPGAs.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Overview rev 1.38 Page 7 of 36 24-Oct-19 2:18 AM Megh Confidential
1.5 The Spark Configuration
The Spark configuration performs the same function as the Native Java configuration. Its purpose is to demonstrate the integration of the Megh Video Analytics Solution on a distributed platform like Spark. The Spark configuration uses the Spark standalone cluster manager.
Consider the Spark configuration in the FPGA mode as shown in Figure 2.
Figure 2: The Spark Configuration Software Stack
As with the Native Java configuration, an RTSP server runs on the driver. Incoming video is decoded by FPGA1 and recognized by FPGA2. The worker node runs a Kafka producer, which includes the RTSP client and the Arka Runtime. The Kafka producer connects to a Kafka broker; there is one broker per worker node. The Kafka broker communicates with the Spark worker, which is the consumer.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 8 of 36 Required Hardware rev 1.38 Megh Confidential
2 Required Hardware
Table 1 lists reference hardware for the demo. The demo configurations described in these release notes contain two servers, called a driver and a worker. The worker node contains two Intel® Arria10 FPGA PACs
Table 1: Required Hardware
Hardware Description
Dell R740
HPE ProLiant DL380 Gen 10
minimum 2 processors Intel® Xeon® Silver 4112 4 cores; 8.25M cache 96 GiB memory or more 1TB storage demo configuration: driver server with one worker server worker server has two system NICs; they must be SFP+ system can be expanded to include additional worker servers.
breakout cable 2x Intel X4DACBL50 cable QSFP+ to 4SFP+ one (1) for each PAC
QSA Adapter 40G QSFP+ to 10G SFP+ Converter Module (10Gtek)
Intel® Arria10 FPGA PAC
2x Intel® Arria10 FPGA PACs (v1.2), each in a PCIe x16 slot demo configuration: two (2) PACs in each worker server none in driver server
Switch 10G switch with a minimum of four (4) ports three (3) ports per node. one (1) port per driver node
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
How to Connect the Single-Worker Demo rev 1.38 Page 9 of 36 24-Oct-19 2:18 AM Megh Confidential
3 How to Connect the Single-Worker Demo
Figure 3 shows the reference hardware configuration for the single-worker demo. Note that this configuration does not require a switch.
The NICs that connect to the user network are called public; the NICS that connect to the FPGAs are called private. The private SFP+ NIC on the driver is assigned the static IP 192.168.10.10; the private SFP+ NIC on the worker is assigned the static IP 192.168.10.21.
The 40G connector of the breakout cable is plugged into PAC 1. The breakout cable has four 10G ports. The 10G Port 1 is connected to the Private NIC on the driver. The 10G Port 2 is connected to the private NIC on the worker; this connection is referred to as the “bump in the wire.” The 10G Port 3 is connected to the 10G connector on the QSA adapter. The 40G connector on the QSA adapter is connected to PAC 2.
Because there is only one worker, this configuration is called a non-distributed configuration. The driver provides the video feed. A typical application might use a video camera to provide the video feed. A distributed configuration would have two or more workers and would require the use of a framework such as Spark.
Figure 3: Demo Configuration with One Worker
Ensure that the PAC firmware is at v1.2. Go to https://www.intel.com/content/www/us/en/programmable/documentation/iyu1522005567196.html for directions on how to identify the PAC firmware version and how to upgrade if necessary.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 10 of 36 How to Connect the Multiple-Worker Demo rev 1.38 Megh Confidential
4 How to Connect the Multiple-Worker Demo
Figure 4 shows the hardware configuration for the demo configuration with multiple workers. An SFP+ transceiver is plugged into the Private NIC of the driver and connects to a 10G switch.
Note that the demo configuration has one driver and two workers. The NICs that connect to the user network are called public; the NICS that connect to the FPGAs are called private. The private SFP+ NIC on the driver is assigned the static IP 192.168.10.10; the private SFP+ NIC on the worker is assigned the static IP 192.168.10.21.
Each worker has two breakout cables. The 40G connectors of one is plugged into PAC 1 and the 40G connector of the other is plugged into PAC 2.
Each breakout cable has four 10G ports. For PAC1, the 10G Port 1 and the 10G Port 3 are connected to the switch. The 10G Port 2 is connected to the private NIC on the worker; this connection is referred to as the “bump in the wire.” For PAC 2, the 10G Port 3 is connected to the switch.
This distributed configuration can be expanded beyond two workers. Each additional worker requires three additional connections to the switch.
In a distributed configuration the driver node provides the video stream and runs the Spark driver. Note, it is possible to run spark with only one worker using the hardware configuration described in Section 3.
Figure 4: Demo Configuration with more than One Worker Ensure that the PAC firmware is at v1.2.. Go to https://www.intel.com/content/www/us/en/programmable/documentation/iyu1522005567196.html for directions on how to identify the PAC firmware version and how to upgrade if necessary.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Required Software (Ubuntu) rev 1.38 Page 11 of 36 24-Oct-19 2:18 AM Megh Confidential
5 Required Software (Ubuntu)
Start with a base Ubuntu 16.04 system that includes all updates. When you install the Megh software packages, the packages check all the required dependencies and install any that are missing.
Table 2: Required Software (Ubuntu)
Software Description
Spark spark-2.4.4-bin-hadoop2.7.tgz Spark Configuration only
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 12 of 36 Required Software (CentOS) rev 1.38 Megh Confidential
6 Required Software (CentOS)
Start with a base CentOS 7 system that includes all updates. When you install the Megh software packages, the packages check all the required dependencies and install any that are missing.
Table 3: Required Software (CentOS)
Software Description
Spark spark-2.4.4-bin-hadoop2.7.tgz Spark Configuration only
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Software Installation (Ubuntu) rev 1.38 Page 13 of 36 24-Oct-19 2:18 AM Megh Confidential
7 Software Installation (Intel)
Workers. This installation is the same for all supported operating systems.
Install the Intel Acceleration Stack Development or Runtime Package on the worker nodes. The worker nodes are the nodes that contain the Intel Programmable Acceleration Cards (PACs). https://www.intel.com/content/www/us/en/programmable/documentation/iyu1522005567196.html#zhm1522795327495
Install the Development package if you plan to do FPGA development.
8 Software Installation (Ubuntu)
Currently, Ubuntu 16.04 with all its latest updates is a supported operating system. This section assumes that the base version of Ubuntu is already installed.
Ensure that Docker is installed on all workers before installing Megh software.
To run the Native Java configuration, it is not necessary to install Spark and Kafka.
Install the RTSP server on the driver. RTSP requires that demodata_0.2.0_noarch.deb also be installed on the driver.
2. Edit /etc/default/grub. Add the kernel parameter (separated by a space from any previous kernel parameters) "hugepagesz=1GB hugepages=2" to GRUB_CMDLINE_LINUX_DEFAULT. Note that there may not be any previous kernel parameters and in that case the line looks as follows: GRUB_CMDLINE_LINUX_DEFAULT="hugepagesz=1GB hugepages=2"
3. Then create the hugetlbfs group, add it to your user’s groups, and update grub. sudo groupadd -g 1006 hugetlbfs sudo usermod -aG hugetlbfs ${USER} sudo update-grub
4. Reboot to update the kernel boot parameters, refresh groups, and mount the hugepages filesystem.
Workers. For both the Native Java Configuration and the Spark Configuration, install all the Megh packages on each worker.
Driver. For both the Native Java Configuration and the Spark Configuration, install megh-vas-demodata-0.2.0_noarch.deb and megh-vas-webui-0.3.0_noarch.deb on the driver.
Driver. For the Spark Configuration, install megh-vas-apps_0.3.0_amd64.deb on the driver.
Assuming that the packages are copied into ~/workspace, issue sudo apt install ./*.deb
If an error occurs when installing these packages, issue sudo dpkg --configure -a
If a wiring configuration change occurs, reconfigure by issuing sudo /var/lib/dpkg/info/megh-vas-bitstreams.postinst
The script postinst rebuilds /opt/megh-vas/scripts/FPGAConf.sh. If you have previously edited FPGAConf.sh and then run postinst, you will lose your edits.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Software Installation (CentOS) rev 1.38 Page 15 of 36 24-Oct-19 2:18 AM Megh Confidential
9 Software Installation (CentOS)
Currently, CentOS 7 is a supported operating system. This section assumes that the base version of CentOS 7 is already installed.
Ensure that Docker is installed before installing Megh software.
To run the Native Java configuration, it is not necessary to install Spark and Kafka.
Install the RTSP server on the driver. RTSP requires that demodata_0.2.0_noarch.rpm also be installed on the driver.
2. Edit /etc/default/grub. Add the kernel parameter (separated by a space from any previous kernel parameters) "hugepagesz=1GB hugepages=2" to GRUB_CMDLINE_LINUX_DEFAULT. Note that there may not be any previous kernel parameters and in that case the line looks as follows: GRUB_CMDLINE_LINUX_DEFAULT="hugepagesz=1GB hugepages=2"
3. Then create the hugetlbfs group, add it to your user’s groups, and update grub. sudo groupadd -g 1006 hugetlbfs sudo usermod -aG hugetlbfs ${USER} sudo grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
4. Reboot to refresh the kernel parameters, update groups, and mount hugepages filesystem.
Workers. For both the Native Java Configuration and the Spark Configuration, install all the Megh packages on each worker.
Driver. For both the Native Java Configuration and the Spark Configuration, install megh-vas-demodata-0.2.0_noarch.rpm and megh-vas-webui-0.3.0_noarch.rpm on the driver.
Driver. For the Spark Configuration, install megh-vas-apps_0.2.0_amd64.rpm on the driver.
1. Assuming that the packages are copied into ~/workspace, issue sudo yum install ./*.rpm
2. After installing these packages issue sudo docker load -i /opt/megh-vas/webui/demo-server.tgz
If a wiring configuration change occurs after installation, rerun the megh-vas-bitstreams_0.3.0_noarch.rpm post install script with: sudo rpm -q --scripts megh-vas-bitstreams |sed '1d' >postinst.sh sudo chmod +x postinst.sh sudo ./postinst.sh
3. The script postinst rebuilds /opt/megh-vas/scripts/FPGAConf.sh. If you have previously edited FPGAConf.sh and then run postinst, you will lose your edits.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Configure FPGAs and NIC rev 1.38 Page 17 of 36 24-Oct-19 2:18 AM Megh Confidential
10 Start the RTSP Server
Driver. Log into the driver node.
cd /opt/megh-vas/video live555MediaServer
You may see several errors displayed. They are benign.
11 Configure FPGAs and NIC
These steps involve editing FPGAConf.sh which is located in /opt/megh-vas/scripts.
1. Select the port that the FPGA uses for DLE input.
If you have set up the configuration shown in Figure 3 (the configuration using the QSA adapter and not the switch), edit FPGAConf.sh to specify UPD0 as follows.
If you have set up the configuration shown in Figure 4 (the configuration using the switch), verify that FPGAconf.sh specifies UPD2 as follows. UDP2 is the default configuration.
2. Specify the IP address of the private NIC on the worker. This is the NIC that configures the “bump in the wire.” If you have more than one worker, do this for each worker. The example line below shows this address as 192.168.10.51. and the NIC interface address as eno1. Set the IP address value for your installation.
ip addr add 192.168.10.51 dev eno1
3. Specify the subnet for your installation in CIDR notation. The example below shows a prefix length of 24 bits and a NIC interface address of eno1. Set it to the value for your installation.
ip route add 192.168.10.0/24 dev eno1
4. Edit the echo statement so that it returns the MAC address of the worker and the IP address of the private NIC on the worker. The example below shows the MAC address as 3c:fd:fe:9c:62:61, the IP address as 192.168.10.51., and the NIC interface address as eno1. Set the IP address value for your installation.
echo "Nic eno1 MAC: 3c:fd:fe:9c:62:61 IP Address: 192.168.10.51"
Run FPGAConf.sh. This script returns the NIC interface address, the MAC address, and the private IP address of the worker. It also returns the two bus IDs for the FPGAs on the worker.
sudo ./FPGAConf.sh Nic enp175s0f1 MAC: 3c:fd:fe:9c:62:61 IP Address: 192.168.10.51 FPGA1 216 FPGA2 134
Remember this information. You will need it when configuring the demo. Make a backup of your edited FPGAConf.sh.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 18 of 36 Configure and Execute the Native Java Demo rev 1.38 Megh Confidential
5. To ensure that the pipeline is set up correctly, execute ping. Specify the IP address of the private NIC on the driver and the NIC the interface address returned by FPGAConf.sh. The example below shows 192.168.10.10 as the IP address of the driver; this may be different for your installation.
ping 192.168.10.10 -I eno1
If ping fails rerun the postinst script (Section 8.4 for Ubuntu; Section 9.4 for CentOS). Edit the new FPGAConf.sh to contain the values in your FPGAConf.sh backup. However, retain the Bus IDs specified in the new FPGAConf.sh.
12 Configure and Execute the Native Java Demo
The Native Java Demo is described in Section 1.4 The Native Java Configuration. There are two pipelines of the Native Java Demo.
The CPU Only pipeline does not make use of the FPGAs. Configuration consists of editing config_cpu.properties.
The CPU+FPGA pipeline does use the FPGAs to perform Video Decode and Image Inference for added performance. Configuration consists of editing config_fpga.properties.
A Web GUI provides visualization of the demo’s execution.
The following sections describe configuring and executing the two pipelines for the Native Java Configuration. This example assumes that you are running both pipelines on the same hardware configuration at the same time.
Although the Spark Configuration provided by Megh uses the Spark standalone cluster manager, when we refer to the Standalone Configuration, we are referring to the Native Java Configuration, not the Spark Configuration. We call the Native Java Configuration the Standalone Configuration because it does not use the Spark framework
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Configure and Execute the Native Java Demo rev 1.38 Page 19 of 36 24-Oct-19 2:18 AM Megh Confidential
12.1 Configure CPU Only Pipeline (Native Java)
1. Workers. Login to worker where the Native Java application is deployed. /opt/megh-vas is the folder where the Native Java application is deployed.
2. Workers. Make following changes to /opt/megh-vas/properties/config_cpu.properties.
Set the RTSP server variables: the MAC address, the IP address, and the port of the private NIC of the driver. For Ubuntu the port number is 8554; for CentOS the port number is 554. Ubuntu is assumed for this example. The MAC and IP addresses are examples; set them to the values for your installation.
#RTSP Server ffmpeg.server.mac=d0:94:66:0e:b1:2a ffmpeg.server.ip=192.168.10.10 ffmpeg.server.port=8554
Set the RTSP client variables: the ffmpeg local client port and the ffmpeg mode.
Set image.path to where the output files are stored, fps.file.path to where FPS (frames/second) data are written, and classifications.file.path to where the video analytics results are written. These locations are a shared mount across all nodes. The shared mount defaults to /opt/megh-vas/resources. Set it to the value for your installation.
IMPORTANT: Ensure that this local client port setting is different from that for the CPU+FPGA mode.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 20 of 36 Configure and Execute the Native Java Demo rev 1.38 Megh Confidential
12.2 Configure CPU+FPGA Pipeline (Native Java)
1. Workers. Login to worker where the Native Java application is deployed. /opt/megh-vas is the folder where the Native Java application is deployed.
2. Workers. Make the following changes to /opt/megh-vas/properties/config_fpga.properties.
Set the RTSP server variables: the MAC address, the IP address, and the port of the private NIC of the driver. For Ubuntu the port number is 8554; for CentOS the port number is 554. Ubuntu is assumed for this example. The MAC and IP addresses are examples; set them to the values for your installation.
#RTSP Server ffmpeg.server.mac=52:54:00:de:47:5e ffmpeg.server.ip=192.168.10.10 ffmpeg.server.port=8554
Set the RTSP client variables: the ffmpeg mode, the MAC address, the IP address, the local port, the remote port.
IMPORTANT: Ensure that this local client port setting is different from that for the CPU ONLY mode.
For fpga0 (out), set the MAC address, the IP address, and the port. For fpga1 (in) set the MAC address, the IP address, and the port. These values are examples; set them to the values for your installation.
Set image.path to where the output files are stored, fps.file.path to where FPS (frames/second) data are written, and classifications.file.path to where the video analytics results are
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Configure and Execute the Native Java Demo rev 1.38 Page 21 of 36 24-Oct-19 2:18 AM Megh Confidential
written. These locations are a shared mount across all nodes. The shared mount defaults to /opt/megh-vas/resources. Set it to the value for your installation.
3. In the terminal window where the CPU+FPGA pipeline is running, execution can be terminated by issuing a CTRL-C.
If the application does not terminate successfully, issue sudo ps ax | grep java
In the displayed list of running processes, identify the process id for the process ending with config_fpga.properties. Then issue the command,
sudo kill -9 <process id>
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 22 of 36 Configure and Execute the Native Java Demo rev 1.38 Megh Confidential
12.5 Start the Web GUI
1. Driver. Set the variable DEMO_DATA to the location of the shared resources directory specified in the properties files. For example, the resources file might be /opt/megh-vas/resources. export DEMO_DATA=/opt/megh-vas/resources
2. Driver. Start Docker. docker-compose -f /opt/megh-vas/webui/docker-compose.yaml up -d
3. Bring up a browser. Specify the public URL of the driver (where Docker is running). Figure 5 shows the Web GUI with the Native Java Configuration.
Figure 5: Native Java Configuration: Web GUI
4. Driver. To bring down the running web application, stop Docker by issuing the command docker-compose -f /opt/megh-vas/webui/docker-compose.yaml down
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Install, Configure, Execute the Spark Demo rev 1.38 Page 23 of 36 24-Oct-19 2:18 AM Megh Confidential
13 Install, Configure, Execute the Spark Demo
Before configuring the Spark Configuration, install and start Spark and Kafka.
The Spark Demo is described in Section 1.5 The Spark Configuration. There are two pipelines for the Spark Demo. Each pipeline has a producer and a consumer.
The CPU Only pipeline does not make use of the FPGAs. The Spark demo has a producer and a consumer. Configuration consists of the following steps:
For the producer, edit ffmpeg_cpu.properties. For the consumer, edit streaming_cpu.properties and Analytics_Zoo_StructuredStream_CPU_VideoAnalytics.sh.
The CPU+FPGA pipeline uses the FPGAs to perform Video Decode and Image Recognition for added performance. The Spark Demo has a producer and a consumer. Configuration consists of the following steps:
For the producer, edit ffmpeg_fpga.properties. For the consumer, edit streaming_fpga.properties and Analytics_Zoo_StructuredStream_FPGA_VideoAnalytics.sh
A Web GUI provides visualization of the demo’s execution.
The following sections describe configuring and executing the two pipelines for the Spark Configuration. This example assumes that you are running both pipelines on the same hardware configuration at the same time.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 24 of 36 Install, Configure, Execute the Spark Demo rev 1.38 Megh Confidential
13.1 Install, Configure, Start Spark
Spark must be installed on the driver and all workers. Spark configuration and execution do not require sudo access.
You may find it useful to open at least two terminal windows and keep them side by side. In one window log into the driver. In the other window log onto the worker.
13.1.1 Install and Configure Spark
Install Spark v2.4.4 (for Hadoop 2.7) on from the link: https://spark.apache.org/downloads.html. The downloaded file is spark-2.4.4-bin-hadoop2.7.tgz. If a later patch version exists, use it.
1. Driver and Workers. On the driver and all workers, copy spark-2.4.4-bin-hadoop2.7 to /opt/spark and extract it. The extraction creates and populates a directory called spark-2.4.4-bin-hadoop2.7.
2. Set the SPARK_HOME environment variable. export SPARK_HOME=/opt/spark/spark-2.4.4-bin-hadoop2.7
3. Enter the directory $SPARK_HOME/conf and copy spark-env.sh.template to spark-env.sh.
4. Edit spark-env.sh to configure Spark for your site This file is sourced when running Spark programs.
Define the Spark Master Host. This is the public IP address of the driver. Set it to the value for your installation. SPARK_MASTER_HOST=<public IP address of the driver>
5. In the directory $SPARK_HOME/conf, copy slaves.template to slaves and append the public IP addresses of all workers (each on a separate line).
6. Workers. Enter the directory $SPARK_HOME/conf and edit spark-env.sh
Define the Spark configuration properties that apply only to the worker. SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.AppDataTtl=172800"
13.1.2 Start Spark
Driver. On the driver node, stop an already running cluster (if there is one) and start a new one. Executing stop-all.sh when the cluster is not running is not an error. Note that although these scripts are executed on the driver they also stop and start Spark
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Install, Configure, Execute the Spark Demo rev 1.38 Page 25 of 36 24-Oct-19 2:18 AM Megh Confidential
13.2 Install, Configure, Start Kafka
Kafka must be installed on the driver and all workers. Download from the link: https://www.apache.org/dyn/closer.cgi?path=/kafka/2.2.0/kafka_2.12-2.2.0.tgz.
1. Driver. Extract the file /opt/kafka/kafka_2.12-2.2.0.tgz into the directory /opt/kafka. The extraction creates a directory called kafka_2.12-2.2.0.
2. Driver. Log into the driver and edit /opt/kafka/kafka_2.12-2.2.0/config/zookeeper.properties and define dataDir as follows: dataDir=/scratch/zookeeper
3. Workers. Extract the file /opt/kafka/kafka_2.12-2.2.0.tgz into the directory /opt/kafka. The extraction creates a directory called kafka_2.12-2.2.0.
4. Workers. Log into each worker and edit /opt/kafka/kafka_2.12-2.2.0/config/server.properties as follows. Add/edit the following lines to the Server Basics section: #port the socket server listens to port=9092 #host the kafka server binds to
host.name=<public IP address of the worker>
Add/edit the following lines to the Socket Server Settings section: listeners=PLAINTEXT://<public IP address of the worker>:9092 advertised.listeners=PLAINTEXT://<public IP address of the worker>:9092
Add/edit the following line to the Log Basics section. Ensure that there is ample disk space. The log file will be large log.dirs=/tmp/kafka-logs
Add/edit the following line to the Log Retention Policy section: log.retention.hours=2
Add/edit the following lines to the Zookeeper section: zookeeper.connect=<public IP address of the driver>:2181
5. Driver. Log in to the driver node and start zookeeper. cd /opt/kafka/kafka_2.12-2.2.0 ./bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 26 of 36 Install, Configure, Execute the Spark Demo rev 1.38 Megh Confidential
6. Workers. Log in to each worker node and start kafka. cd /opt/kafka/kafka_2.12-2.2.0 ./bin/kafka-server-start.sh -daemon ./config/server.properties
Create topics and partitions. This example assumes a two-node system. We have one worker and one broker per worker. Hence, we show the replication-factor as 1.
./bin/kafka-topics.sh --create --zookeeper <public IP address of the driver>:2181 \ --topic imagestream-cpu --partitions <spark executor core number; we use 8> \ --replication-factor <equivalent to number of brokers; we use 1> ./bin/kafka-topics.sh --create --zookeeper <public IP address of the driver>:2181 \ --topic imagestream-fpga --partitions <spark executor core number; we use 8> \ --replication-factor <equivalent to number of brokers; we use 1>
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Install, Configure, Execute the Spark Demo rev 1.38 Page 27 of 36 24-Oct-19 2:18 AM Megh Confidential
13.3 Configure CPU Only Pipeline (Spark Demo)
13.3.1 Producer Configuration (CPU Only Spark Demo)
1. Workers. Log in to the worker node. The /opt/megh-vas is the folder where the Spark application is deployed.
2. Workers. Make following changes to /opt/megh-vas/properties/ffmpeg_cpu.properties.
Set the RTSP server variables: the MAC address, the IP address, and the port of the private NIC of the driver. For Ubuntu the port number is 8554; for CentOS the port number is 554. Ubuntu is assumed for this example. The MAC and IP addresses for ffmpeg.server are examples; set them to the values for your installation.
##KAFKA CONFIG #Variable
kafka.bootstrap.servers=<public IP address of the worker>:9092 #Topic name topics=imagestream-cpu #Set to broker id as set in /opt/kafka/kafka_2.12-2.2.0/config/server.properties broker.id= #RTSP Server ffmpeg.server.mac=d0:94:66:0e:b1:2a ffmpeg.server.ip=192.168.10.10 ffmpeg.server.port= 8554
Set the RTSP client variables: the ffmpeg local client port and the ffmpeg mode.
ffmpeg.local.port=11000 ffmpeg.mode=CPU
Set image.out.folder to where the output files are stored. This location is a shared mount across all nodes. The shared mount defaults to /opt/megh-vas/resources. Set it to the value for your installation.
IMPORTANT: Ensure that this local client port setting is different from that for the CPU+FPGA mode.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 28 of 36 Install, Configure, Execute the Spark Demo rev 1.38 Megh Confidential
13.3.2 Consumer Configuration (CPU Only Spark Demo)
The consumer configuration described here uses a fixed spark configuration. The example specifies the number of executor cores and driver and executor memory. Experiment with and modify these for improved performance
1. Driver. Log into the driver and make the following changes to /opt/megh-vas/properties/streaming_cpu.properties #Variable Kafka Config
bootstrap.servers=<public IP address of the worker>:9092
2. Driver. Edit the file /opt/megh-vas/scripts/Analytics_Zoo_StructuredStream_CPU_VideoAnalytics.sh
as follows:
#SPARK CONFIGS #Spark driver IP
SPARK_HOST=<public IP address of the driver> #Set to spark home directory. export SPARK_HOME=/opt/spark/spark-2.4.4-bin-hadoop2.7
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Install, Configure, Execute the Spark Demo rev 1.38 Page 29 of 36 24-Oct-19 2:18 AM Megh Confidential
13.4 Configure CPU+FPGA Pipeline (Spark Demo)
This section describes the CPU+FPGA pipeline for the Spark Configuration.
This configuration involves editing the files ffmpeg_fpga.properties and server.properties.
1. Workers. Log into the worker node. /opt/megh-vas is the folder where the Spark application is deployed.
2. Workers. Make the following changes to /opt/megh-vas/properties/ffmpeg_fpga.properties.
## KAFKA CONFIG #Variable #Set to executor/worker IP
kafka.bootstrap.servers=<public IP address of the worker>:9092
#Set to broker id as set in /opt/kafka/kafka_2.12-2.2.0/config/server.properties broker.id=0 #Topic name topics=imagestream-fpga
Set the RTSP server variables: the MAC address, the IP address, and the port of the private NIC of the driver. For Ubuntu the port number is 8554; for CentOS the port number is 554. Ubuntu is assumed for this example. The MAC and IP addresses are examples; set them to the values for your installation.
#RTSP Server ffmpeg.server.mac=d0:94:66:0e:b1:2a ffmpeg.server.ip=192.168.10.10 ffmpeg.server.port=8554
Set the ffmpeg local client port.
#RTSP Client
Recall the values returned by FPGAConf.sh. Do not run FPGAConf.sh again. The values shown below are example values. Use the values from your own installation.
#FPGA NIC ffmpeg.local.mac=3c:fd:fe:9c:62:61
IMPORTANT: Ensure that this local client port setting is different from that for the CPU Only mode.
Nic enp175s0f1 MAC: 3c:fd:fe:9c:62:61 IP Address: 192.168.10.51 FPGA1 216 FPGA2 134
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 30 of 36 Install, Configure, Execute the Spark Demo rev 1.38 Megh Confidential
For fpga0 (out), set the MAC address, the IP address, and the port. For fpga1 (in) set the MAC address, the IP address, and the port. These values are examples; set them to the values for your installation.
#VD FPGA Out ffmpeg.fpga0.out.mac=00:11:22:33:44:72 ffmpeg.fpga0.out.ip=192.168.10.72 ffmpeg.fpga0.out.port=9568 #DLE FPGA In ffmpeg.fpga1.in.mac=00:11:22:33:44:73 ffmpeg.fpga1.in.ip=192.168.10.73 ffmpeg.fpga1.in.port=9569
Set the Bus IDs. They must be in decimal. FPGAconf.sh returns the values in decimal. The values shown below are example values. Use the values returned by FPGAConf.sh.
ffmpeg.bus.id.0=216 ffmpeg.bus.id.1=134
Set image.out.folder to where the output files are stored. This location is a shared mount across all nodes. The shared mount defaults to /opt/megh-vas/resources. Set it to the value for your installation.
#The folder must be named "images_fpga" image.out.folder=/opt/megh-vas/resources/images_fpga/
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Install, Configure, Execute the Spark Demo rev 1.38 Page 31 of 36 24-Oct-19 2:18 AM Megh Confidential
The consumer configuration described here uses a fixed spark configuration. The example specifies the number of executor cores and driver and executor memory. Experiment with and modify these for improved performance.
This configuration involves editing the files streaming_fpga.properties and Analytics_Zoo_StructuredStream_FPGA_VideoAnalytics.sh.
1. Driver. Log in to the driver node and make the following changes to /opt/megh-vas/properties /streaming_fpga.properties.
Set to where the output files are stored. This location is shared mount across all nodes. This shared mount defaults to /opt/megh-vas/resources.
#Variable Output Config classification.out.file=/opt/megh-vas/resources/classifications_fpga.sock fps.out.file=/opt/megh-vas/resources/fps_fpga.sock set bootstrap server #Variable Kafka Config #Set to executor/worker iP
bootstrap.servers=< public IP address of the worker>:9092
2. Driver. Make the following changes to /opt/megh-vas/scripts/Analytics_Zoo_StructuredStream_FPGA_VideoAnalytics.sh #SPARK CONFIGS #Spark driver IP
SPARK_HOST=<public IP address of the driver>
Environment Config #Set to spark home directory export SPARK_HOME=/opt/spark/spark-2.4.4-bin-hadoop2.7
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 32 of 36 Install, Configure, Execute the Spark Demo rev 1.38 Megh Confidential
13.5 Execute the Spark Demo
13.5.1 Producer Execution (Spark CPU Only Demo)
1. Workers. Log in to each worker and run the JavaCV CPU producer. cd /opt/megh-vas/scripts/ sudo ./StartJavaCVProducer-CPU.sh
2. In the terminal window where the producer program is running, execution can be terminated by issuing a CTRL-C.
13.5.2 Consumer Execution (Spark CPU Only Demo)
1. Driver. Log into the driver and run the CPU consumer. cd /opt/megh-vas/scripts/ sudo ./Analytics_Zoo_StructuredStream_CPU_VideoAnalytics.sh
2. In the terminal window where the consumer is running, execution can be terminated by issuing a CTRL-C.
13.5.3 Producer Execution (Spark CPU+FPGA Demo)
1. Workers. Log in to each worker node and run the JavaCV FPGA producer. cd /opt/megh-vas/scripts/ sudo ./StartStructuredProducerJavaCV-FPGA.sh
2. In the terminal window where the CPU+FPGA Spark producer is running, execution can be terminated by issuing a CTRL-C. If the application does not terminate successfully, issue ps ax | grep java
In the displayed list of running processes, identify the process id for the process ending with ffmpeg_fpga.properties. Then, issue the command, sudo kill -9 <process id>
13.5.4 Consumer Execution (Spark CPU+FPGA Demo)
1. Driver. Log in to the driver node and run the CPU consumer. cd /opt/megh-vas/scripts sudo ./Analytics_Zoo_StructuredStream_FPGA_VideoAnalytics.sh
2. In the terminal window where the consumer is running, execution can be terminated by issuing a CTRL-C.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Install, Configure, Execute the Spark Demo rev 1.38 Page 33 of 36 24-Oct-19 2:18 AM Megh Confidential
13.5.5 Start the Web GUI
1. Driver. Set the variable DEMO_DATA to the location of the resources directory. export DEMO_DATA=/opt/megh-vas/resources
2. Driver. Start Docker. sudo docker-compose -f /opt/megh-vas/webui/docker-compose.yaml up -d
3. Bring up a browser. Specify the URL of the driver (where Docker is running). Figure 6 shows the Web GUI with the Spark Configuration.
Figure 6: Spark Configuration: Web GUI 4. Driver. To bring down the running web application, stop Docker by issuing the command
sudo docker-compose -f /opt/megh-vas/webui/docker-compose.yaml down
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 34 of 36 Performance rev 1.38 Megh Confidential
14 Performance
The Alpha 2 release of the Megh Video Analytics Solution is the first fully functional pipeline demonstrating video decode + image recognition offload to FPGA. The CPU+FPGA pipeline can sustain eight (8) channels of 1080p@30 fps each. This release does not have the hooks to measure pipeline latency, but this feature is planned for a future release.
The following is a list of some AFU implementation details.
Video Decode AFU on FPGA1.
Implements a single channel UDP filtering, RTP decap, H.264 decode, YUV-to-RGB color space conversion and Image Resizing.
Deep Learning Engine on FPGA2
Implements channel multiplier and Deep Learning Engine configured for SqueezeNet at 240 fps (the DLE can actually achieve 4000 fps) to match Video Decode pipeline performance. Future releases on Intel PAC A10 will support a choice of Image Recognition and Object Detection topologies.
The Alpha 2 release of the Megh Video Analytics Solution utilizes only 35% of the resources on each FPGA, allowing for the possibility of future scale-out.
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
Known Issues rev 1.38 Page 35 of 36 24-Oct-19 2:18 AM Megh Confidential
15 Known Issues
Table 4: Known Issues
Issue Description Impact
1 When running both the CPU Only and the CPU+FPGA pipelines simultaneously in either the Native Java or Spark configuration, you may see the error
Bind failed: Address already in use
The system outputs NaN values.
Workaround
Ensure that both the CPU Only and CPU+FPGA configurations have different settings for ffmpeg.local.port.
2 After configuring the FPGA bitstreams and subsequently running the CPU+FPGA pipeline in either the Native Java or Spark configuration, the data does not stream.
This issue does not always occur.
The second pipeline that was started will not run. The VD engine hangs.
Workaround
For the CPU+FPGA Native Java configuration, terminate execution by issuing a CTL-C. Then re-launch by executing
cd /opt/megh-vas/scripts sudo ./Standalone-FPGA-Headless-Mode.sh
For the CPU+FPGA Spark configuration, terminate producer execution and consumer execution by issuing a CTL-C in the windows where they were started.
Then re-launch the producer by executing cd /opt/megh-vas/scripts sudo ./StartStructuredProducerJavaCV-FPGA.sh
In another window, re-launch the consumer by executing cd /opt/megh-vas/scripts sudo ./Analytics_Zoo_StructuredStream_FPGA_VideoAnalytics.sh
MEGH COMPUTING, INC., 1600 NE Compton Drive, Suite 202, Hillsboro, OR 97006
24-Oct-19 2:18 AM Page 36 of 36 Known Issues rev 1.38 Megh Confidential
Issue Description Impact
3 During operation, the application may experience an intermittent timeout
The application hangs.
Workaround
The same as for Issue 2.
4 Failure on worker with message: Connection to <…>?timeout=0 failed: No route to host
The worker fails to pull RTSP data from the driver.
Workaround
Install RTSP on the workers. For a CentOS driver testRTSPClient rtsp://<private IP address of the driver>:554/Utube_Long_LowRes_Video.264 For an Ubuntu driver testRTSPClient rtsp://<private IP address of the driver>:8554/Utube_Long_LowRes_Video.264
If the above test fails, On the CentOS driver sudo systemctl stop firewalld.service On an Ubuntu driver sudo service ufw stop
![Megh Boleche Jabo Jabo by Humayun Ahmed[Part[1].1]](https://static.documents.pub/doc/80x56/55292ad94a795977158b4656/megh-boleche-jabo-jabo-by-humayun-ahmedpart11.jpg)
![Megh Boleche Jabo Jabo by Humayun Ahmed[Part.2].PDF](https://static.documents.pub/doc/80x56/563dbb60550346aa9aaca20b/megh-boleche-jabo-jabo-by-humayun-ahmedpart2pdf.jpg)
