TeraVM Architecture
TeraVM system level architecture Architecture Document
Introduction
© Shenick Network Systems - 2 -
TeraVM Architecture
1 Introduction ............................................................................................................................................... 3
2 Client Interfaces ........................................................................................................................................ 4
2.1 Java GUI .............................................................................................................................................................5
2.2 CLI ......................................................................................................................................................................6
2.3 TCL API ..............................................................................................................................................................8
2.4 iTest Plugin .........................................................................................................................................................8
2.5 Web Interface .....................................................................................................................................................9
3 TeraVM Controller .................................................................................................................................. 10
3.1 Management Network ....................................................................................................................................... 10
3.2 PostgreSQL Database ...................................................................................................................................... 11
3.3 Java Server....................................................................................................................................................... 11
3.4 Mongo DB ......................................................................................................................................................... 13
3.5 Statistics SRRD ................................................................................................................................................ 13
3.6 Apache httpd with PHP ..................................................................................................................................... 14
3.7 Configuration Files ............................................................................................................................................ 14
4 TeraVM Test Module ............................................................................................................................... 14
4.1 Communication or “Comms” Network................................................................................................................ 14
4.2 MLIPS (Test Agent) ......................................................................................................................................... 15
4.3 TeraVM-1 vs. TeraVM-2 .................................................................................................................................... 15
4.4 Fedora Guest OS .............................................................................................................................................. 16
5 License Server ........................................................................................................................................ 16
Introduction
© Shenick Network Systems - 3 -
TeraVM Architecture
1 Introduction
This document assumes a working knowledge of how a TeraVM system is used to perform L2 –
L7 testing and how a basic system is configured. For those that are not familiar or would
otherwise like a comprehensive review we suggest reading or at least scanning the product user
guide (also referred to as the “diversifEye User Guide”).
Because we will refer to these 3 items in the document we provide below a brief description of
each. For a more detailed understanding please refer to the product user guide.
1. Test Group: A test group is basically a definition of a test case and is the high level
construct that a TeraVM user begins with when defining test cases. A test group contains
multiple entities such as hosts and applications. Only one instance of a test group can be
run by a user at a given time.
2. Host: In a test group each host represents a network node (e.g. mobile phone, laptop,
set top box etc.) in the emulated network. Basically any network node that has L2 – L3
properties (i.e. MAC address, IP address) is represented in TeraVM as a host.
3. Application: Just as in a real world scenario applications are deployed on hosts (e.g.
mobile phone, laptop). Within a TeraVM system users define applications (e.g. HTTP,
VoIP) as per their test requirements and subsequently assign those defined applications
to one or more hosts which they have previously defined.
Just as many network orientated software applications can be running on a single
networked PC at any one time (e.g. web browser, email client), many applications can
run on the same TeraVM host at the same time.
This document provides a comprehensive architectural overview of the entire TeraVM system.
Every major building block in each subsystem is shown with enough description for a reasonably
technical person to grasp how the building blocks interwork and contribute to the overall system.
There are 4 major subsystems: 1) Client Interfaces, 2) TeraVM Controller 3) TeraVM Test
Modules and 4) TeraVM License Server as shown in Figure 1. Next we will discuss each
subsystem in detail.
Client Interfaces
© Shenick Network Systems - 4 -
Figure 1: TeraVM Overall Architecture
2 Client Interfaces
The client interfaces subsystem provides the various mechanisms by which a human or machine
(i.e. test automation) can interact with a TeraVM system to build and execute test cases. The
client interfaces subsystem only communicates with a TeraVM Controller and does so specifically
using a Java RMI (Remote Method Invocation) interface as shown in Figure 1. RMI is
a Java API that performs the object-oriented equivalent of remote procedure calls (RPC), with
support for direct transfer of serialized Java objects and distributed garbage collection.
Client Interfaces
© Shenick Network Systems - 5 -
Figure 2: Client Interface Options
Below we will describe each component listed in Figure 2.
2.1 Java GUI
The Java based graphical user interface (GUI) is the main vehicle used for human interaction with
TeraVM. The client provides an interface for configuring, executing, viewing and analyzing L2 –
L7 test scenarios and the statistics gathered during the execution of these test scenarios. The
client must be downloaded and installed on a local machine such as a laptop and is referred to as
the “diversifEye Client” as it was originally designed and deployed for interacting with a
diversifEye chassis or appliance.
There is a detailed description of the features and functions of the Java GUI client (“diversifEye
Client”) beginning with chapter 2 of the product user guide (also referred to as the “diversifyEye
User Guide”).
Client Interfaces
© Shenick Network Systems - 6 -
Figure 3: Java GUI Client
2.2 CLI
The command line interface (CLI) is the most often used mechanism to automate and script
interaction with a TeraVM Controller. The CLI is unique to TeraVM but uses a conventional verb-
noun style format. Some example CLI commands are as follows:
Log Out of Cli
$> exit
Configure Cli username and partition
$> cliConfigure
Add a test folder
$> cli addTestGroupFolder "//" <FolderName> <This_is_a_description>
List available tests
$> cli listTestGroups
Client Interfaces
© Shenick Network Systems - 7 -
Create a testgroup in a folder
$> cli addTestGroupFolder
Import a TestGroup
$> cli importTestGroup "//myfolder/ or //" <testname.xml>
Start a TestGroup
$> cli startTestGroup <testname>
Stop a TestGroup
$> cli stopTestGroup <testname>
Delete a TestGroup
$> cli deleteTestGroup <testname>
2.2.1 Perl XML Generator
Shenick does not provide a Perl API per se. When most people hear the term “Perl API” they
think we provide a Perl library of functions which they can call directly from a Perl script. The
expectation is that these library functions will mimic what we can do from the CLI, namely
functions such as starting and stopping tests, and capturing test output in real time.
Instead what Shenick provides is a PERL library which allows a user to define a test case with all
relevant test groups, hosts, applications etc. The defined test cases can then be output in XML
format and read in via the CLI and ultimately become part of the PostgreSQL database which
persistently stores all available test groups. This functionality is very helpful but requires multiple
manuals steps (i.e. export XML, load via CLI) and also does not lend itself to real time (i.e.
start/stop tests) scripting of a TeraVM system.
Client Interfaces
© Shenick Network Systems - 8 -
Figure 4: Perl XML Generation Example
2.3 TCL API
The TCL API is built on top of the Java API and is of course exposed through the TCL scripting
language. Most of the functionality supported by the Java API is exposed via TCL with similar
methods and parameters.
There is not a large customer user base for the TCL API as the CLI and Perl XML Generator are
more feature rich. As a result, minimal engineering effort has been put in to enhancing the TCL
API in recent years.
2.4 iTest Plugin
iTest is an integrated test authoring and execution solution from Spirent Communications
(acquisition of a startup called Fanfare) which provides an SDK to enable 3rd parties to develop
plug-ins for iTest allowing it to access and control their products. This interface is not used by
very many Shenick customers and details of configuration can be found in the “diversifEye User
Guide” chapter 16 entitled “Fanfare Integration”.
Client Interfaces
© Shenick Network Systems - 9 -
2.5 Web Interface
Figure 5: TeraVM Web Interface
The TeraVM web interface serves many different purposes, but it is in some sense the first place
to go after installing a TeraVM Controller. Access is very simple and just involves browsing to the
IP address of the TeraVM Controller. For our example in Figure 5 it would be simply
http://192.168.21.67.
As show in Figure 1 the TeraVM Controller has an Apache web server running in it which of
course hosts the web interface.
Following are some other functions of the web interface
Download Java GUI client
Backup and restore configurations
Upgrade to another version of TeraVM
Download system logs
Install CLI and/or scripting environments (e.g. TCL, PERL, Java API)
Global Settings
The last item, global settings, serves another very significant purpose for the web interface.
TeraVM Controller
© Shenick Network Systems - 10 -
Figure 6: Web Interface Global Settings Page
There are a wide variety of global settings that are made from this page. These include specific
settings for many of the supported applications such as TCP, HTTP, RTSP, and TWAMP as well
as cryptographic, IPv6 and general network settings to name but a few. Some of the settings are
temporary and just added in until the function can be formally introduced in to the Java GUI client.
Other settings are for customer specific implementations and hence are not to be exposed to a
wider audience. In general, adding a setting to the web interface is a quick way to add a capability
for proof of concept or time to market purposes. As shown in Figure 1 all settings that are made
via the web interface are committed to “Configuration Files” in the TeraVM Controller that are
ultimately read by the “Test Execution Server” when loading or executing a test.
3 TeraVM Controller
A significant part of the TeraVM Controller is a Java server which contains a variety of
components including the Test Execution Server which is the main vehicle for test execution and
communication with TeraVM Test Modules and the License Server.
3.1 Management Network
The TCP network between the client interfaces and TeraVM Controller is referred to as the
“management” network because it is the means by which a user “manages” their TeraVM
deployment. This network can have either a private (RFC 1918) or public IP addressing scheme
depending upon how the end user wants to access the TeraVM Controller. The IP address of the
TeraVM Controller
© Shenick Network Systems - 11 -
TeraVM Controller that is used by the Java GUI Client and Web Interface for access to the
Controller is clearly part of the management network.
Figure 7: TeraVM Controller
3.2 PostgreSQL Database
Client interface defined tests are persistently stored in this database. In other words if a user
defines test groups with various hosts and applications via the Java GUI client or CLI the entire
configuration is stored persistently in this database.
Whenever a test is run the “Test Execution Server” pulls the test configuration information (e.g.
hosts, applications etc.) from this database via the persistence layer.
3.3 Java Server
A significant portion of the TeraVM Controller is the Java Server which is often referred to as the
“back-end” because it serves as the back end for all of the client interfaces.
3.3.1 Test Execution Server
The Test Execution Server (TES) is in some sense the brains of the TeraVM Controller. When a
TeraVM user runs a test this triggers the TES to load, create and then execute the test. There are
6 key steps involved in this scenario as follows:
TeraVM Controller
© Shenick Network Systems - 12 -
Step 1: The defined test configuration is read from the PostgreSQL database and additional data
is taken in from relevant configuration files
Step 2: The test configuration is expanded into the run time datastore, which is backed by
MongoDB
Step 3: The licenses required to run the desired test are checked out from the license server
Step 4: Test agent(s) are instantiated on the appropriate test module(s) based on the test
configuration
Step 5: Statistics SRRD is informed that test agent(s) have been instantiated and will be sending
statistics updates directly to SRRD. This action triggers SRRD to establish a TCP connection with
each instantiated test agent (that it was just informed about) in order to receive statistics updates.
Step 6: If thresholds have been set, TES starts the thresholding engine
3.3.2 Thresholding Engine
Thresholding is a powerful TeraVM capability. Here is a brief description of the capability from the
product user guide.
“Thresholding functionality is available via the GUI and CLI and allows an operator to
automatically determine if something has gone wrong during the execution of the test. If an
operator is interested in knowing if, for example, the Mean Get Time of any HTTP Client
download is greater than 1s, a threshold containing this criteria can be applied to all HTTP Clients
and an event will be generated if the criteria is breached. This capability prevents the operator
from having to sift through lots of statistics to see if something has gone wrong.”
The Thresholding Engine receives a threshold filter (e.g. “show me every HTTP client with a
mean get time greater than 1s”) from a client interface (e.g. Java GUI or CLI) and passes it to the
“Statistics SRRD” system. This triggers SRRD to begin sifting through all the statistics returned
from the relevant test agents and filter out those that meet the filter criteria. This match data is
then sent to the Thresholding Engine and in turn forwarded to the client interface subsystem for
display or output.
3.3.3 Service Layer
The service layer is the private development API to the Java Server. It exposes all test
management functionality including test creation/editing, and test execution. It also exposes
secondary functionality around tests such as packet captures and statistics. The server layer is
exposed to the client interfaces via RMI.
TeraVM Controller
© Shenick Network Systems - 13 -
3.3.4 Manager Layer
The manager layer is an intermediate layer between the service layer and the persistence layer.
A manager exists for each test entity type. It is here that any business logic is implemented. It is
generally a light layer with very little code.
3.3.5 Persistence Layer
The persistence layer is an abstraction layer between the database and the rest of the server. It
takes care of the persistence of entities in either the static test database (PostgreSQL) and the
run time test database (MongoDb)
3.4 Mongo DB
This is a run time database that is utilized during test execution to persist the state of the running
entities (e.g. hosts, applications). It also persists run time assigned properties such as
dynamically assigned IPs (i.e. via DHCP), MTU, and gateway address. During test execution
certain information about the test is pulled from Mongo DB to update client interfaces.
When a test is stopped all related information is deleted from Mongo DB. This is therefore a
temporary, run time database.
Mongo DB is particularly useful for scaled entity testing because a large amount of information
must be manipulated to emulate tens of thousands of entities or more.
3.5 Statistics SRRD
This is the statistics engine which contains a database that is referred to as Simple Round Robin
Database or SRRD. Simply put this is where all test related statistics or metrics are stored and
retrieved. The entire thresholding feature depends upon SRRD to keep track of statistics which
are collected during test execution. For example if a user is interested in knowing if the “Mean
Get Time” of any HTTP Client download is greater than 1s, a threshold containing this criteria can
be applied to all HTTP Clients and an event will be generated if the criteria is breached.
SRRD establishes a separate TCP connection to each TeraVM Test Agent and this is the
mechanism by which it receives constant, real time statistics or metrics updates from each test
module. The Test Execution Server tells SRRD that a given test agent has been instantiated and
this triggers SRRD to establish a TCP connection with that test agent for statistics collection
purposes.
TeraVM Test Module
© Shenick Network Systems - 14 -
3.6 Apache httpd with PHP
This is an Apache web server daemon (httpd) with the PHP module loaded. PHP is used to
dynamically generate the web interface pages and to write to specific configuration files.
3.7 Configuration Files
These configuration files contain information that is set in the Web Interface for example global
settings.
Here are some examples of the configuration files that are updated:
/opt/shenick/diversifEye/master-server/MasterServerX.props
/etc/shenick/partitions.cfg
/etc/shenick/UI-defaults.props
/etc/mlips.conf
4 TeraVM Test Module
TeraVM Test Modules are the virtual machines that support the entities which actually generate
the traffic which is used to test a device or network under test. In some sense the test module is
the reason why a customer buys TeraVM and hence when we generically refer to “TeraVM” we
are referring to the actual test modules and not the controller or anything else. A “TeraVM
Controller” is named that way because it controls a “TeraVM” or in other words it controls one or
more TeraVM Test Modules.
Figure 7 shows three TeraVM Test Modules and how they interface with a TeraVM Controller. In
a typical deployment there are many more than three TeraVM Test Modules.
4.1 Communication or “Comms” Network
The TCP network between the TeraVM Controller and TeraVM test modules is often referred to
as the “communication” or “comms” network. Simply put the “comms” network is a private
network between all TeraVM test modules and the TeraVM Controller they are controlled by. A
given TeraVM test module can only be associated with one TeraVM Controller. The “comms”
network should have a private (RFC 1918) IP addressing scheme (e.g. 192.168.0.0/16) to avoid
any conflict with externally routable IPs. By default the subnet of 192.168.240.0/20 is used for
“comms” but this is user configurable. In the default set up IP address 192.168.240.1 is assigned
to the Controller, 192.168.240.2 is reserved (for potential future use) and the first TeraVM test
module will use IP 192.168.240.3 and so on until 192.168.255.254. This provides for a maximum
of 4,091 TeraVM test modules that can be controlled by a single TeraVM Controller.
TeraVM Test Module
© Shenick Network Systems - 15 -
4.2 MLIPS (Test Agent)
The traffic used to test a device or network under test is generated by one or more software
instances running inside a Test Module. These software instances are called the test agent. The
TeraVM test agent is also known as MLIPs (Multiple Lightweight TCP/IP Stack Simulator)
MLIPS is without a doubt Shenick’s “crown jewels” and the holder of much of Shenick’s
intellectual property.
Simply put, the MLIPS application is used by a TeraVM system to generate Layer 4-7 TCP and
UDP traffic over IP and provide other functionality to facilitate the running of TeraVM tests.
MLIPS is sometimes referred to as a “packet processor” or “PP”.
MLIPS has its own highly customized IP stack that is based upon New Reno.
We sometimes generically refer to MLIPS as the “test agent” or in other words the agent that is
generating the tests.
Although Figure 7 shows the test module with only one instance of MLIPS there is no reason why
a single test module cannot contain multiple instances of MLIPS or in other words multiple test
agents.
4.3 TeraVM-1 vs. TeraVM-2
We often refer to a TeraVM-2 or TVM-2 for short. This is a single test module or VM appliance
(TeraVM application + Guest OS) that utilizes a total of 2 CPU cores. MLIPS runs on one CPU
core and the other is used for interrupt processing. By contrast a TeraVM-1 (TVM-1) uses only
one core (running MLIPS) and has no interrupt core.
TVM-1 is useful for test scenarios where we are willing to sacrifice performance and deterministic
results in favor of testing with twice as many VMs and vNICs for the same amount of CPU
resources. An example would be vSwitch testing where the task is to send small amounts of data
across thousands of vNICs.
In the future Shenick might build a TVM-5 or TVM-8 which could contain multiple instances of
MLIPS each running on their own CPU core and some number of extra CPU cores reserved for
interrupt processing.
License Server
© Shenick Network Systems - 16 -
4.4 Fedora Guest OS
A TeraVM Test Module is in fact a virtual appliance by which we mean it is a package that
contains both the TeraVM software as well as a specific guest operating system. This package
can be deployed on supported hypervisors such as ESXi, Hyper-V and KVM.
In the case of a TeraVM Test Module the underlying guest OS is Fedora. Fedora is a well-known
open source Linux operating system that is supported by the Fedora project and Red Hat
Corporation. One of Fedora's main objectives is not only to contain software distributed under
a free and open source license, but also to be on the leading edge of such technologies. Fedora
developers prefer to make upstream changes instead of applying fixes specifically for Fedora—
this ensures that their updates are available to all Linux distributions. Red Hat Corporation
actually branches their Red Hat Enterprise Linux (RHEL) releases from versions of Fedora.
Fedora became the TeraVM Test Module guest OS in release 10.3. Prior to that the guest OS
was BusyBox which is a very stripped down and hence lightweight Linux OS. One of the main
reasons for switching to Fedora was so we could have a “mainstream” guest OS with support for
all common test and debugging tools. BusyBox apparently had limited tools support.
5 License Server
The TeraVM License Server (TMV-LS) is the only subsystem of a TeraVM system that is not
developed by Shenick. TVM-LS is developed by Reprise Software which includes the team at
Reprise that created the most widely-deployed license manager, FLEXlm. Shenick purchases the
product directly from Reprise and provides it to TeraVM customers.
TVM-LS cannot be administered from any of the available TeraVM client interfaces because it is
a third party product and has not been integrated from an administration perspective. All TVM-LS
administration is done from an independent web interface shown in Figure 8.
License Server
© Shenick Network Systems - 17 -
Figure 8: TVM-LS Administrative Web Interface (Reprise Software)
As shown in Figure 7 the only direct interaction from TVM-LS is via a TCP connection to the “Test
Execution Server”.
A TVM-LS licenses file is a human readable text file that contains signatures to prevent
unauthorized editing.
Following is a sample snippet from a license file:
-----------------------------------------------
HOST localhost 005056875de8
ISV shenick
LICENSE shenick rlm_server_enable_vm 1.0 permanent 1 _ck=87d6fc7a6a
sig="60P04504ARQFQ855M5C5HB9YN0Q5NSWH9C89NVR22GMAMRDQY8QG0VPYQP8J0V
6J0T3PYBYH9C"
LICENSE shenick tvmc 1.0 permanent 1000 timezone=ffffff _ck=f86670c509
sig="60Q04580RWCT94HKANRXHJ7WBMGA4NB2S6V1XAPE08AG16VW580D2EDGU0VMC3
WV9STFRQXCT94QA"
License Server
© Shenick Network Systems - 18 -
LICENSE shenick tvm_cores 1.0 permanent 1000 timezone=ffffff _ck=106670089c
sig="60P0451E358NPEASNYCHMRNU7AJE4K01F9YMVG822HDWVTD68V9PTT4D0CR3CW6
T9C6CX43PA4"
LICENSE shenick esxi_5.0_tvm_cores 1.0 permanent 1000 _ck=1a66fcec8f
sig"60PG4580MT502QG84UU909ST1YX860VD0FM6Y7PN08A30PW85ABDJX96FAGTW0G
G0MT709047BA0"
-----------------------------------------------------------------------
TeraVM Architecture
© Shenick Network Systems. All rights reserved. Shenick Network Systems, Shenick, the Shenick logo, TeraVM, and
diversifEye are registered trademarks of Shenick Network Systems, Inc. All other products and companies may be
trademarks of their respective owners.
www.shenick.com
NORTH AMERICA
1900 McCarthy Blvd
Suite 301
Milpitas, CA 95035
TEL : 408-385-7630
EUROPE
Brook House, Corrig Avenue
Dun Laoghaire, Dublin, Ireland
TEL : +353-1-236-7002