Date post: | 14-Jan-2016 |
Category: |
Documents |
Upload: | leon-lambert |
View: | 213 times |
Download: | 0 times |
MIT Lincoln LaboratorySPP-HPEC09-1BAM 04/21/23
A Special-Purpose Processor System with Software-Defined
Connectivity
Benjamin Miller, Sara Siegal, James Haupt, Huy Nguyen and Michael Vai
MIT Lincoln Laboratory
22 September 2009
This work is sponsored by the Navy under Air Force Contract FA8721-05-0002. Opinions, interpretations , conclusions and recommendations are those of the authors and are not necessarily endorsed by the United States Government.
MIT Lincoln LaboratorySPP-HPEC09-2BAM 04/21/23
Outline
• Introduction
• System Architecture
• Software Architecture
• Initial Results and Demonstration
• Ongoing Work/Summary
MIT Lincoln LaboratorySPP-HPEC09-3BAM 04/21/23
Why Software-Defined Connectivity?
• Modern ISR, COMM, EW systems need to be flexible
– Change hardware and software in theatre as conditions change
– Technological upgrade– Various form factors
• Want the system to be open– Underlying architecture
specific enough to reduce redundant software development
– General enough to be applied to a wide range of system components
E.g., different vendors
• Example: Reactive electronic warfare (EW) system
– Re-task components as environmental conditions change
– Easily add and replace components as needed before and during mission
MIT Lincoln LaboratorySPP-HPEC09-4BAM 04/21/23
Antenna Interface
Special Purpose Processor (SPP) System
• System representative of advanced EW architectures
– RF and programmable hardware, processors all connected through a switch fabric
LN
A
RF
Dis
trib
uti
on Rx 1
Rx 2
Rx R
RF
Dis
trib
uti
on
HP
A
FPGA 1 Processor 1
Enabling technology: bare-bone, low-latency pub/sub
middleware
Enabling technology: bare-bone, low-latency pub/sub
middleware
FPGA M
FPGA 2 Processor 2
Processor N
Switch Fabric
Thin Communications Layer (TCL) Middleware
Config. Mgr. Res. Mgr.
Switch Fabric
Proc. 1 Proc. 2OS
FPGA 1 FPGA M Rx RRx 1 Tx TTx 1
data processingalgorithms
OSProc. N
OS
Process PProcess 2
Process 1
configurebackplane
connections
manageinter-processconnections
Tx 1
Tx 1
Tx T
MIT Lincoln LaboratorySPP-HPEC09-5BAM 04/21/23
Mode 1: Hardwired
• Hardware components physically connected
• Connections through backplane are fixed (no configuration management)
• No added latency but inflexible
Thin Communications Layer (TCL) Middleware
Config. Mgr. Res. Mgr.
Switch Fabric
Proc. 1 Proc. 2OS
FPGA FPGA
RxRx TxTx
OSProc. NOS
FPGA FPGA
Process PProcess 2Process 1
connections tohardware fixed
MIT Lincoln LaboratorySPP-HPEC09-6BAM 04/21/23
Mode 2: Pub-Sub
• Everything communicates through the middleware– Hardware components have on-board processors running proxy
processes for data transfer
• Most flexible, but there will be overhead due to the middleware
ProcOS
Process 4
FPGA
Thin Communications Layer (TCL) Middleware
Res. Mgr.
Proc ProcOS FPGA OS
ProcOS
Rx
Switch Fabric
FPGATx Proc
OS Rx
Process 3Process 2Proxy 1
Process PProcess 2
Process 1
MIT Lincoln LaboratorySPP-HPEC09-7BAM 04/21/23
Mode 3: Circuit Switching
• Configuration manager sets up all connections across the switch fabric
• May still be some co-located hardware, or some hardware that communicates via a processor through the middleware
• Overhead only incurred during configuration
Thin Communications Layer (TCL) Middleware
Config. Mgr. Res. Mgr.
Switch Fabric
Proc. 1 Proc. 2OS
FPGA 1 FPGA M Rx RRx 1 Tx TTx 1
OSProc. NOS
Process PProcess 2
Process 1
MIT Lincoln LaboratorySPP-HPEC09-8BAM 04/21/23
Today’s Presentation
• TCL middleware developed to support the SPP system– Essential foundation
• Resource Manager sets up (virtual) connections between processes
Thin Communications Layer (TCL) Middleware
Config. Mgr. Res. Mgr.
Switch Fabric
Proc. 1 Proc. 2OS
FPGA 1 FPGA M Rx RRx 1 Tx TTx 1
OSProc. NOS
Process PProcess 2
Process 1
MIT Lincoln LaboratorySPP-HPEC09-9BAM 04/21/23
Outline
• Introduction
• System Architecture
• Software Architecture
• Initial Results and Demonstration
• Ongoing Work/Summary
MIT Lincoln LaboratorySPP-HPEC09-10
BAM 04/21/23
System Configuration
• 3 COTS boards connected through VPX backplane
– 1 Single-board computer, dual-core PowerPC 8641
– 1 board with 2 Xilinx Virtex- 5 FPGAs and a dual-core 8641
– 1 board with 4 dual-core 8641s– Processors run VxWorks
• Boards come from same vendor, but have different board support packages (BSPs)
• Data transfer technology of choice: Serial RapidIO (sRIO)
– Low latency important for our application
• Implement middleware in C++
MIT Lincoln LaboratorySPP-HPEC09-11
BAM 04/21/23
BSP2
VPX + Serial Rapid I/OVPX + Serial Rapid I/O
Physical Interface
Operating System (Realtime: VxWorks; Standard: Linux)VxWorks
BSP1
System Model
HW (Rx/Tx, ADC, etc.)
VendorSpecific
ApplicationComponents
Signal Processing Lib
System ControlComponents
MIT Lincoln LaboratorySPP-HPEC09-12
BAM 04/21/23
VPX + Serial Rapid I/O
Operating System (Realtime: VxWorks; Standard: Linux)
System Model
VendorSpecific
VPX + Serial Rapid I/O
VxWorks
BSP1 BSP2 HW (Rx/Tx, ADC, etc.)
TCL Middleware
Signal Processing Lib
System ControlComponents
ApplicationComponents
Physical Interface
MIT Lincoln LaboratorySPP-HPEC09-13
BAM 04/21/23
BSP1 BSP2
System Model
VPX + Serial Rapid I/OVPX + Serial Rapid I/O
Physical Interface
Operating System (Realtime: VxWorks; Standard: Linux)VxWorks
HW (Rx/Tx, ADC, etc.)
VendorSpecific
TCL Middleware
ApplicationComponents
ApplicationComponents
System ControlComponents
Signal Processing Lib
New components can easily be added by complying with the middleware API
New SW components
New HWcomponents
MIT Lincoln LaboratorySPP-HPEC09-14
BAM 04/21/23
Outline
• Introduction
• System Architecture
• Software Architecture
• Initial Results and Demonstration
• Ongoing Work/Summary
MIT Lincoln LaboratorySPP-HPEC09-15
BAM 04/21/23
Switch Fabric
Publish/Subscribe Middleware
Process l1Process k
pu
blish
Process l2
Proc. l1 Proc. l2
Topic T Subscribers------------------
l1l2
Publishing application doesn’t need to know where the data is going . . .
. . . and the subscribers are unconcerned about where their data comes from
send toapplication
del
iver
deliver
TCL Middleware
Middleware acts as interface to both application and hardware/OS
Proc. k
no
tify
no
tify
OS OSOS
send
to l1
send
to l2
subscribers
MIT Lincoln LaboratorySPP-HPEC09-16
BAM 04/21/23
Abstract Interfaces to Middleware
• Middleware must be abstract to be effective– Middleware developers are unaware of hardware-specific libraries– Users have to implement functions that are specific to BSPs
TCL Middleware
publisher/subscribermgmt
interface with application
interface with
hardware/OS
arrivalnotification send data to
subscribers
accept data from
publishers
How (exactly) do I execute a data transfer?
What data-transfer
technology am I using?
MIT Lincoln LaboratorySPP-HPEC09-17
BAM 04/21/23
XML Parser
• Resource manager is currently in the form of an XML parser– XML file defines topics, publishers, and subscribers– Parser sets up the middleware and defines virtual network
topology
Thin Communications Layer (TCL) Middleware
Config. Mgr. Res. Mgr.
Switch Fabric
Proc. 1 Proc. 2
OS
FPGA 1 FPGA M Rx RRx 1 Tx TTx 1
OSProc. N
OS
Process PProcess 2
Process 1
setup.xml Parser
MIT Lincoln LaboratorySPP-HPEC09-18
BAM 04/21/23
Builder
has ahas a
CustomBSP-Builder
de
riv
ed
fro
m
Interfaceto BSP
Middleware Interfaces
• Base classes– DataReader, DataReaderListener and DataWriter
interface with the application– Builder interfaces with BSPs
• Derive board- and communication-specific classes
has aDataReader
DataReaderListener
DataWriter
de
riv
ed
fro
m
SrioData-Reader
SrioData-ReaderListener
SrioData-Writerd
eri
ve
d f
rom
de
riv
ed
fro
m
Interface to Application
change with hardware
change with comm. tech.
has a
MIT Lincoln LaboratorySPP-HPEC09-19
BAM 04/21/23
Builder
CustomBSP-Builder
de
riv
ed
fro
m
Interfaceto BSP
Builder
• Follows the Builder pattern in Design Patterns*
• Provides interface for sRIO-specific tasks– e.g., establish sRIO connections, execute data transfer
• Certain functions are empty (not necessarily virtual) in the base class, then implemented in the derived class with BSP-specific libraries
#include <math.h>#include “vendorPath/bspDma.h”...//member functions STATUS CustomBSPBuilder::performDmaTransfer(...){ return bspDmaTransfer(...);}...
#include <math.h>... //member functions STATUS Builder::performDmaTransfer(...){}...
*E. Gamma et. al. Design Patterns: Elements of Reusable Object-Oriented Software. Reading, Mass.: Addison-Wesley, 1995.
MIT Lincoln LaboratorySPP-HPEC09-20
BAM 04/21/23
has aDataReader
DataReaderListener
DataWriter
der
ived
fro
m
SrioData-Reader
SrioData-ReaderListener
SrioData-Writerd
eriv
ed f
rom
der
ived
fro
m
Interface to Application
Publishers and Subscribers
• DataReaders, DataWriters and DataReaderListeners act as “Directors” of the Builder
– Tell the Builder what to do, Builder determines how to do it
• DataWriter used for publishing, DataReader and DataReaderListener used by subscribers
• Derived classes implement communication(sRIO)-specific, but not BSP-specific, functionality
– e.g., ring a recipient’s doorbell after transferring data
//member functionsvirtual STATUS DataWriter::write(message)=0;
virtual STATUS SrioDataWriter::write(message){... myBuilder->performDmaXfer(…);...}
Derived Builder type determined dynamically
MIT Lincoln LaboratorySPP-HPEC09-21
BAM 04/21/23
Outline
• Introduction
• System Architecture
• Software Architecture
• Initial Results and Demonstration
• Ongoing Work/Summary
MIT Lincoln LaboratorySPP-HPEC09-22
BAM 04/21/23
Software-Defined ConnectivityInitial Implementation
<Topic><Name>Send</Name><ID>0</ID><Sources> <Source>
<SourceID>8</SourceID> </Source>
</Sources><Destinations> <Destination>
<DSTID>0</DSTID> </Destination></Destinations>
</Topic><Topic>
<Name>SendBack</Name><ID>1</ID><Sources> <Source>
<SourceID>0</SourceID> </Source>
</Sources><Destinations> <Destination>
<DSTID>8</DSTID> </Destination></Destinations>
</Topic>
<Topic><Name>Send</Name><ID>0</ID><Sources> <Source>
<SourceID>8</SourceID> </Source>
</Sources><Destinations> <Destination>
<DSTID>0</DSTID> </Destination></Destinations>
</Topic><Topic>
<Name>SendBack</Name><ID>1</ID><Sources> <Source>
<SourceID>0</SourceID> </Source>
</Sources><Destinations> <Destination>
<DSTID>8</DSTID> </Destination></Destinations>
</Topic>
• Experiment: Process-to-process data transfer latency
– Set up two topics– Processes use TCL to
send data back and forth– Measure round trip time
with and without middleware in place
Process 2
TCL Middleware
Process 1
Switch Fabric
Proc. 1 Proc. 2OS
FPGA 1 FPGA M Rx RRx 1 Tx TTx 1
OSProc. N
OS
MIT Lincoln LaboratorySPP-HPEC09-23
BAM 04/21/23
Software-Defined ConnectivityCommunication Latency
• One-way latency ~23 us for small packet sizes
• Latency grows proportionally to packet size for large packets
• Reach 95% efficiency at 64 KB
• Overhead is negligible for large packets, despite increasing size
P1 P2
MIT Lincoln LaboratorySPP-HPEC09-24
BAM 04/21/23
Demo 1: System Reconfiguration
Processor #1 Processor #2 Processor #3
Detect
Detect Transmit Process
TCL Middleware
Process Transmit
Configuration 1:
Configuration 2:
XML1
XML2
Objective: Demonstrate connectivity reconfiguration by simply replacing the configuration XML file
Detect Process Transmit
Detect Transmit Process
MIT Lincoln LaboratorySPP-HPEC09-25
BAM 04/21/23
Transmit Proc #2
Receive Proc #1
Low-latency predefined connections allow quick response
Demo 2: Resource Management
Receive Proc #1 Transmit
TCL Middleware
Proc #2Control
I’ve detectedsignals!
I will process this newinformation!
MIT Lincoln LaboratorySPP-HPEC09-26
BAM 04/21/23
Transmit Proc #2
Receive Proc #1
Demo 2: Resource Management
I need more help toanalyze the signals!
I will determine what totransmit in response!
Resource manager sets up new connections on demand to efficiently utilize available computing power
Receive Proc #1 Transmit
TCL Middleware
Proc #2Control
Working…
MIT Lincoln LaboratorySPP-HPEC09-27
BAM 04/21/23
Transmit Proc #2
Receive Proc #1
Demo 2: Resource Management
Working…
Proc #1/Transmit are publisher/subscriber on topic TransmitWaveform
I will transmit theresponse!
I have determined anappropriate response!
Receive Proc #1 Transmit
TCL Middleware
Proc #2Control
MIT Lincoln LaboratorySPP-HPEC09-28
BAM 04/21/23
Transmit Proc #2
Receive Proc #1
Demo 2: Resource Management
Done!
After finishing, components may be re-assigned
Receive Proc #1 Transmit
TCL Middleware
Proc #2Control
I will transmit theresponse!
I have determined anappropriate response!
MIT Lincoln LaboratorySPP-HPEC09-29
BAM 04/21/23
Outline
• Introduction
• System Architecture
• Software Architecture
• Initial Results and Demonstration
• Ongoing Work/Summary
MIT Lincoln LaboratorySPP-HPEC09-30
BAM 04/21/23
Ongoing Work
Thin Communications Layer (Pub/Sub)
InterferenceMitigation
Software Modules
SW Module Timing
SW Module
ImageFormation
SW Module
Detection
SW ModuleFPGA
FPGA
FFTDIQ
DAC
ADC
IP Cores
FPGAModules
Control
Data PathPhysical Layer (e.g., Switch Fabric)
• Develop the middleware (configuration manager) to set up fixed connections
– Mode 3: Objective system
• Automate resource management- Dynamically reconfigure
system as needs change- Enable more efficient use of
resources (load balancing)
Process PProcess 2
Thin Communications Layer (TCL) Middleware
Config. Mgr. Res. Mgr. Process 1
Switch Fabric
Proc. 1 Proc. 2OS
FPGA 1 FPGA M Rx RRx 1 Tx TTx 1
OSProc. N
OS
MIT Lincoln LaboratorySPP-HPEC09-31
BAM 04/21/23
Summary
• Developing software-defined connectivity of hardware and software components
• Enabling technology: low-latency pub/sub middleware– Abstract base classes manage connections between nodes– Application developer implements only system-specific send and
receive code
• Encouraging initial results– At full sRIO data rate, overhead is negligible
• Working toward automated resource management for efficient allocation of processing capability, as well as automated setup of low-latency hardware connections