Legend:
L. M. Russo, Beam Diagnostics GroupBrazilian Synchrotron Light Laboratory, LNLS, Brazil
THPHA149
The Brazilian Synchrotron Light Laboratory (LNLS) is in the final stages of developing an open-source BPM system for Sirius, a 4th-generation synchrotron light source under construction in Brazil. The system is based on the MicroTCA.4 standard comprising AMC FPGA boards carrying FMC digitizers and a CPU module. The software is built with the HALCS framework and employs a service-oriented architecture (SOA) to export a flexible interface between the gateware modules and its clients, providing a set of loosely-coupled components favoring reusability, extensibility and maintainability. In the paper, the BPM system will be discussed in detail focusing on how specific functionalities of the system are integrated and developed in the framework to provide SOA services. In particular, two domains will be covered: (i) gateware modules, such as the ADC interface, acquisition engine and digital signal processing; (ii) software services counterparts, showing how these modules can interact with each other in a uniform way, easing integration with control systems.
Software and Gatew are Development for Sirius BPM Ele ctronics Using a
Service-Orient ed Architecture
BPM Project Gateware
BPM Project Software using HALCS framework
Protocol Conversion
Peripheral #1
Peripheral #2
Bridge #1
Peripheral #3.1
Peripheral #3.2
Peripheral #4
Communication Interface
Figure 1. Generic Hardware Architecture.
Communication Interface
Peripheral Controller #4
Peripheral Controller #3.2
Peripheral Controller #3.1
External protocol
Intra-Controller protocol
Peripheral Controller #2
Peripheral Controller #1
Figure 2. SOA-based Software Architecture.
Figure 6. BPM Software Architecture.
Client Layer
Broker Layer
Service Layer
Device Access Layer
HALCS DataflowIntroduction
Pipe Protocol
MLM Protocol
Board Support
HAL
Dispatch Engine
Broker
MLM Protocol
Kernel
PCIe
LLIO
DEVIO
SMIO #2 (e.g., Data Acquisition)
SMCH
SMPRSMPR
SMCH
SMIO #1 (e.g., FMC130 Control)
MALAMUTE Broker
Client App #2 (e.g., CLI interface)
Client App #1 (e.g., EPICS IOC)
Eth (TCP)
Figure 3. HALCS Framework Architecture.
LinksHALCS Framework: https://github.com/lnls-dig/halcsBPM EPICS IOC: https://github.com/lnls-dig/bpm-epics-iocBPM Gateware: https://github.com/lnls-dig/bpm-gw DSP Cores: https://github.com/lnls-dig/dsp-cores Infra Cores: https://github.com/lnls-dig/infra-coresTiming Receiver: https://github.com/lnls-dig/timing-receiver-gw
Summary● SOA principles applied to low-level software maximize reuse of commonly used
functionalities● Best used in conjunction with isolated hardware components with minimal interaction
among each other● Succesfully deployed in the BPM and the upcoming MicroTCA.4 Timing Receiver
projects
Generic Hardware Architecture
SOA-based Software Architecture
● Standard communication interfaces: PCIe, UART, etc.
● Hierarchical Design, e.g. based on open-source Wishbone Bus Protocol
● Peripherals with minimal interaction, acting as isolated components
● Desirable to have knowledge about internal components such as: unique ID, name, address range, version, capabilities
● Software abstracts hardware components as services
● Uses a common protocol to communicate with hardware device
● Services can coordinate themselves by using an Intra-Controller protocol
● Protocols acts as a flexible/extensible API
● HALCS Framework implements SOA principles, using an Inversion of Control design paradigm
● Uses a common RPC protocol to expose services functionalities
● Broker provides discoverability and reliability to services using Mailbox messaging pattern
● Services (SMIO) register functions
● Services can use additional abstractions:
● SMCH for external chips: AD9510 clock distributor and PLL, Si57x clock oscillator, etc.
● SMPR for external protocols: SPI, I2C, etc.
● DEVIO: Event-driven reactor engine
● LLIO: Hardware Abstaction Layer
● Reusable gateware components, based on Wishbone B4
● Self-Describing Bus (SDB) aware components:
● Easier for software to be gateware-agnostic: dynamic offsets, version, capabilities
● Software can act as a userspace driver
● Application logic pushed to Client Layer
Figure 5. BPM Hardware Architecture.
Switch. Clock
DSPData,
Controls & Clock
cos
-sin
I
Q
NCO
CORDIC (Rect-Pol)
tbt amp.
CIC
CIC
CIC
CIC
tbt
tbt
CORDIC (Rect-Pol)
fofb amp.
fofb
fofb
CICmon. amp.
x4
DSP Data
Signal Deswap
x2
x2
FMC ADC Board
RFFE Board
LEDsTrigger
Backplane
Addr. SpaceWB Slave
DAQ
Addr. SpaceWB Slave
Trigger Mux
WB SlaveAddr. Space
WB SlaveAddr. Space
Switching Clk. Gen.
WB Master
WB Crossbar
SDB
WB Slave
To all WB Slaves
Addr. SpaceWB Slave
Heartbeat & LEDs
Addr. SpaceWB Slave
Trigger Interface
delay
diff. sum
adc
tbt
fofb
mon.
adc
tbt
fofb
mon.
xyq
sum
fofb amp. 1 to 4
Data
MMC
Addr. SpaceWB Slave
Diagnos- tics
Serial
Addr. Space
PCIe Interface
WB MasterAddr. Space
UART (optional)
WB Master
FPGA
DDR3 SDRAM
2GB
MGT Transceiver
WB SlaveAddr. Space
FMC ADC Interface
WB SlaveAddr. Space
A. Sp.WB Slv.
ADC Clock
A. Sp.WB Slv.
FMC Misc.
1. Caller instantiates LLIO
2. Caller instantiates DEVIO
3. Caller registers SMIOs manually or via SDB
4. SMIOs registers into Malamute Broker
5. Clients may send/receive messages to any SMIO
MSGPIPE
DEVIO PIPE DEVIO PIPE: REGISTER_SMIOREGISTER_SMIO_ALLUNREGISTER_SMIOUNREGISTER_SMIO_ALL
MSG PIPE: External Protocol
Event loop
CMD PIPE: $TERM
MALAMUTE:RPC protocol
Event loop
RPC over Malamute
CMDPIPE
RPC over Malamute
RPC over Malamute
Figure 4. HALCS Dataflow.
HALCS Architecture
Device Interface
RPC Client Interface
Trigger Control & Status
Trigger Interface
Device Interface
RPC Client Interface
Heartbeat&LEDs
LEDs Control & Status
Device Interface
RPC Client Interface
IPMI InfoBoard Status
Diagnostics
x2Device Interface
RPC Client Interface
Testing Functions
LEDsTrigger Control
FMC Status
FMC Misc.
x2Device Interface
RPC Client Interface
DSP Configuration
Swithching Control
DSP
Device Interface
RPC Client Interface
Trigger Mux
Mux Control
x2
Device Interface
Support Functions
Status
Data Transfer
Trigger Control
RPC Client Interface
DAQ
x2
I2C
Oscillator Control Control &
StatusSPI
PLL Control
Device Interface
RPC Client Interface
FMC ADC Clock
x2x2Device Interface
RPC Client Interface
ADC Control
SPI
Control & Status
FMC ADC
Dispatch Engine (DEVIO)
SDB lib
Device Interface
RPC Client Interface
Deswap Control
Deswap
x2
External protocol
Intra-Controller protocol
RPC client protocol
Application-Specific Module (SMIO)SMIO
HAL (LLIO)
PCIe
To all RPC Client Interfaces
To all RPC Client Interfaces
To all Clients
MALAMUTE BrokerTo all Clients
EPICS DriverCommand-Line
Interface
Client Layer
Broker Layer
Service Layer
Device Access Layer