Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | gilbert-goodman |
View: | 235 times |
Download: | 7 times |
Synchronous Device Interface at NSLS-II
Yuke Tian Control Group, NSLS-II, BNL
(May 1, 2009 EPICS Collaboration Meeting, Vancouver)
Outline
1. Synchronous, deterministic and reliable data distribution
2. Synchronous Device Interface
3. Synchronous Device Interface Extension
4. MicroTCA
5. Summary
EPICS Collaboration Meeting, Vancouver 2009
Synchronous, deterministic and reliable data distribution
Requests to control systems from accelerator physics
Distribute data in accelerator complex with:
● Synchronism: Data acquisition and distribution systems run off a single master oscillator.
● Deterministic : Data distribution with a fixed delay.
● High reliability: Data distribution with redundancy, no single point of failure.
NSLS-II Accelerator Complex
EPICS Collaboration Meeting, Vancouver 2009
• Goal: run all circuitry in the accelerator off a single master oscillator
• Advantages
Accelerator are driven by many subsystems (RF, magnet/PS, diagnostics etc). Global synchronism offers repeatable, stable behavior. For example, fast orbit feedback.
Synchronous stimulus-response measurement (such as response matrix, tune measurement etc) are easy in a fully synchronous system.
Global synchronism offers global timestamp.
There is no added jitter from crude local clock.
Synchronous, deterministic and reliable data distribution: synchronism
EPICS Collaboration Meeting, Vancouver 2009
Synchronous, deterministic and reliable data distribution: synchronism
EPICS Collaboration Meeting, Vancouver 2009
Clock Domains In Accelerators
Synchronous Device Interface: challenges for FOFB
EPICS Collaboration Meeting, Vancouver 2009
Synchronous Device Interface: answers for NSLS-II FOFB
EPICS Collaboration Meeting, Vancouver 2009
Synchronous Device Interface: cell controller
EPICS Collaboration Meeting, Vancouver 2009
• Synchronism in NSLS-II
125MHz clock (in sync with 500 RF master clock) and fiducia will be distributed.
We already have a very fine, low jitter clock. We can synchronize all circuit with this RF clock. We can timestamp every event with 2ns resolution. All the system are talking with the same clock.
We will push the synchronism into power supply control system. Then, all subsystem (BPMs, cell controller, PS control etc) will be fully synchronous with RF clock. This will provide a lot of benefits for accelerator physics.
Synchronous Device Interface: synchronism
EPICS Collaboration Meeting, Vancouver 2009
Synchronous Device Interface: deterministic
Data per BPM Data per cell Total data to be distributed
GTP 1.25Gbps 8-bit datawidth
GTP 2.5Gbps 16-bit datawidth
10 bytes(X:4; Y: 4; Status:2)
120 Byte(8 BPM4 XBPM)
120 * 30 = 3.6Kbyte
3.6Kbye/1Gbps= 28.8 us
FPGA: 125MHZ
3.6Kbye/2Gbps=14.4 us
FPGA: 125MHZ
Latency calculations:
EPICS Collaboration Meeting, Vancouver 2009
Synchronous Device Interface: deterministic
GTP interface test: 1.25Gbps 8-bit datawidth
EPICS Collaboration Meeting, Vancouver 2009
Synchronous Device Interface: deterministic
GTP interface test: 2.5Gbps 16-bit datawidth
EPICS Collaboration Meeting, Vancouver 2009
Synchronous Device Interface: reliability
• Synchronous Device Interface reliability:
There are two 2.5Gbps links running in opposite directions. One is the main link and the other is backup link.
When a link (or a pair of links) between two cell controllers is broken, the backup link is used to keep connectivity between any pair of cell controllers in the ring. The system has localized the broken link and re-configured automatically to maintain connectivity using backup link.
The BPM data distribution is abstracted from upper application levels, and therefore corrector setting calculations are not altered when the system falls into the fault recovery operation mode. The fault is identified by the system, and error flags are delivered to the operator level.
EPICS Collaboration Meeting, Vancouver 2009
Synchronous Device Interface: extension
EPICS Collaboration Meeting, Vancouver 2009
Synchronous Device Interface: Extension to PS Control
EPICS Collaboration Meeting, Vancouver 2009
MicroTCA
EPICS Collaboration Meeting, Vancouver 2009
In 1979, Motorola was developing their new Motorola 68000 CPU and one of their engineers, Jack Kister, decided to set about creating a standardized bus system for 68000-based systems, which he called VERSAbus. http://en.wikipedia.org/wiki/VMEbus
Request from distributed control/computing system(looks familiar ?)
MicroTCA
EPICS Collaboration Meeting, Vancouver 2009
EPICS Collaboration Meeting, Vancouver 2009
ANSI/VITA 1.180 MB/s
ANSI/VITA 1.5320+ MB/s VITA 41
3 to 30 GB/sVME with
Switch Fabric on P0
VITA 46VME and/or
Switch Fabrics3U & 6U
IEEE101440 MB/s
VME…VME64…VME64x…VME2eSST…VXS…
1981 1994 1997 2003 2005
VME…VME64…VME64x…VME2eSST…VXS…
1981 1994 1997 2003 2005
Ethernet…10BASE5…10BASE2…10BASE-T…100BASE-T…1000BASE-X…10GBASE-X…
1982 1983 1984 1990 1994 1998 2002
Ethernet…10BASE5…10BASE2…10BASE-T…100BASE-T…1000BASE-X…10GBASE-X…
1982 1983 1984 1990 1994 1998 2002
MicroTCA
EPICS Collaboration Meeting, Vancouver 2009
VME backplane evolution
MicroTCA
How about a cleaner solution ?
MicroTCA
EPICS Collaboration Meeting, Vancouver 2009
Kontron OM9140
Emerson OM5080
Schroff MicroTCA crate Elma Blu!Smart
MicroTCA
EPICS Collaboration Meeting, Vancouver 2009
Technology Bus Bandwidth CPU OS Cost Vendors
VME Parallel+ Serial
VME64: 80MB/s PowerPC LinuxVxWorksRTEMS
Crate: $3-7KCPU: $2-5K
CPU: EmersonCrates: Wiener, Dawn, Elma, Schroff, Rittal
MicroTCA Serials (GigE.PCIe,
RapidIO)
With PCIex41GB/s
Intel multi corePowerPC
LinuxWindows
WindRiver?
$5K (crate, MCH,
and CPU board)
CPU: Emerson, Kontron, GE Fanuc, moreCrates: Dawn, Elma, Schroff, Rittal
Compare of VME and MicroTCA
Challenges of MicroTCA for accelerator control community: 1) IO modules2) Software development.
MicroTCA
Summary
• SDI status:
LBNL group (Larry Doolittle group) is leading the SDI core design. The communication through fiber links are done.
BNL group is designing the SDI extension at PS control system.
Get BPM data into cell controller by using Libera grouping features is under development.
We are evaluating MicroTCA IOC. It will be test on EPICS/Linux. Then we will try to migrate to EPICS/RTEMS if necessary.
EPICS Collaboration Meeting, Vancouver 2009
Summary
• SDI features:
Use the same clock to synchronously distribute data around nodes;
Two directions transmitter and receiver to make it redundant and single node/connection fail safe;
Serial chain link requires minimum wiring;
Protocol independent of carrier/link speed: Rocket IO 2.5Gbps fiber link; 100Mbps Ethernet link;
• SDI will be an open source design.
EPICS Collaboration Meeting, Vancouver 2009
Acknowledgement
Larry Doolittle (LBNL)Bob Dalesio (BNL)
Carlos Serrano (LBNL)Joseph Mead (BNL)
Thank you !
EPICS Collaboration Meeting, Vancouver 2009