Date post: | 27-Mar-2015 |
Category: |
Documents |
Upload: | sara-weeks |
View: | 214 times |
Download: | 1 times |
ESLEA and HEP’s Work ESLEA and HEP’s Work on UKLight Networkon UKLight Network
ESLEAESLEA
Exploitation of Switched Lightpaths in E-Exploitation of Switched Lightpaths in E-sciences Applicationssciences Applications
Multi-disciplinedMulti-disciplined Protocol DevelopmentProtocol Development Exploitation by HEP ( ATLAS and CDF), Radio-Exploitation by HEP ( ATLAS and CDF), Radio-
astronomers, E-Health, HPC astronomers, E-Health, HPC Using dedicated point-to-point light path Using dedicated point-to-point light path
channels on channels on researchresearch UKLight Network for UKLight Network for R&DR&D purposes purposes
Bulk Data Transfers / Circuit Reservation and Bulk Data Transfers / Circuit Reservation and deployment /Transport Protocols / Real Time deployment /Transport Protocols / Real Time VisualizationVisualization
HEP ConnectionsHEP Connections
RAL-CERNRAL-CERN UCL-FermilabUCL-Fermilab Lancaster-EdinburghLancaster-Edinburgh RAL-LancasterRAL-Lancaster SARA-Lancaster SARA-Lancaster Lancaster-ManchesterLancaster-Manchester
Lancaster <-> Edinburgh Lancaster <-> Edinburgh ObjectivesObjectives
Investigate the use of an alternate (in this Investigate the use of an alternate (in this case UDT) protocol to maximise the case UDT) protocol to maximise the potential of an optical circuitpotential of an optical circuit
Utilise this protocol in such a was as to be Utilise this protocol in such a was as to be of practicable use to users of the grid.of practicable use to users of the grid.
What is UDT ?What is UDT ? UDT: UDT: UUDP-based DP-based DData ata TTransfer Protocolransfer Protocol Application level, end-to-end, unicast, Application level, end-to-end, unicast,
reliable, connection-oriented, data transport reliable, connection-oriented, data transport protocol.protocol.
Approximately 90% utilisationApproximately 90% utilisation
of available bandwidthof available bandwidth
ServersServers
Hardware :Hardware : Dual Xeon 3.2GHz dual coreDual Xeon 3.2GHz dual core 2 GB RAM2 GB RAM Dual PCI-X busDual PCI-X bus 2 x Gigabit Ethernet2 x Gigabit Ethernet SATA Raid controllerSATA Raid controller 6 x SATA disks6 x SATA disks 1 x SATA system disk1 x SATA system disk
OSOS Scientific Linux 3.0.5 with 2.4.21Scientific Linux 3.0.5 with 2.4.21
Disk Testing ResultsDisk Testing Results
Ext3 Write Ext3 Write Speed / CPU Speed / CPU LoadLoad
Ext3 Read Ext3 Read Speed / CPU Speed / CPU LoadLoad
XFS Write XFS Write Speed / CPU Speed / CPU LoadLoad
XFS Read XFS Read Speed / CPU Speed / CPU LoadLoad
System DiskSystem Disk 320 Mb/s320 Mb/s
(20%)(20%)
336 Mb/s336 Mb/s
(10%)(10%)-- --
Raid 0 (write Raid 0 (write caching caching disabled)disabled)
1544 Mb/s1544 Mb/s
(96%)(96%)
1304 Mb/s1304 Mb/s
(33%)(33%)
2160 Mb/s2160 Mb/s
(63%)(63%)
1304 Mb/s1304 Mb/s
(34%)(34%)
Raid 0 (write Raid 0 (write caching caching enabled)enabled)
1143 Mb/s1143 Mb/s
(70%)(70%)
1221 Mb/s1221 Mb/s
(30%)(30%)
1184 Mb/s1184 Mb/s
(33%)(33%)
1124 Mb/s1124 Mb/s
(28%)(28%)
Raid 5 (write Raid 5 (write caching caching disabled)disabled)
90 Mb/s90 Mb/s
(6%)(6%)
363 Mb/s363 Mb/s
(10%)(10%)
97 Mb/s97 Mb/s
(3%)(3%)
348 Mb/s348 Mb/s
(10%)(10%)
Raid 5 (write Raid 5 (write caching caching enabled)enabled)
905 Mb/s 905 Mb/s
(60%)(60%)
1222 Mb/s1222 Mb/s
(30%)(30%)
1001 Mb/s1001 Mb/s
(30%)(30%)
1216 Mb/s1216 Mb/s
(30%)(30%)
Network TestingNetwork Testing Tests were performed with default kernel Tests were performed with default kernel
and application settings and then again and application settings and then again after applying changes to maximise after applying changes to maximise network speedsnetwork speeds
BDP for this link should be :BDP for this link should be : BDP = Bandwidth (MB/s) * RTT (seconds)BDP = Bandwidth (MB/s) * RTT (seconds) BDP = (1 * 1024 / 8) * (0.3 / 1000)BDP = (1 * 1024 / 8) * (0.3 / 1000) BDP = 0.0384 MB (39.32KB)BDP = 0.0384 MB (39.32KB)
Basic Network Test resultsBasic Network Test results
ProtocolProtocol ““Out of the box” Out of the box” throughput throughput
Maximum ThroughputMaximum Throughput
TCPTCP 650 Mb/s650 Mb/s 940 Mb/s940 Mb/s
UDPUDP 957 Mb/s957 Mb/s 957 Mb/s957 Mb/s
UDTUDT 950 Mb/s950 Mb/s 950 Mb/s950 Mb/s
File Transfer Test ResultsFile Transfer Test Results
Transfer Transfer ProgramProgram
““Out of the box” tuningOut of the box” tuning Maximal ThroughputMaximal Throughput
ftpftp 565 Mb/s565 Mb/s 820 Mb/s820 Mb/s
GridFTPGridFTP 570 Mb/s570 Mb/s 850 Mb/s850 Mb/s
UDT test UDT test applicationapplication 460 Mb/s460 Mb/s 935 Mb/s935 Mb/s
What next ?What next ?
The Basic Network tests and the File The Basic Network tests and the File transfer tests need to be re-performed transfer tests need to be re-performed once the UKLight link between Lancaster once the UKLight link between Lancaster and Edinburgh is fully functionaland Edinburgh is fully functional
Integration of UDT into a functional Integration of UDT into a functional GridFTP server and clientGridFTP server and client
Deployment of modified software into test Deployment of modified software into test LCG sites.LCG sites.
Lancaster<->RAL LinkLancaster<->RAL Link
T1-T2 transfer testingT1-T2 transfer testing Avoid production network induced Avoid production network induced
bottlenecksbottlenecks Firewall @ RALFirewall @ RAL Internal LAN trafficInternal LAN traffic
Tested using :Tested using : Command line srmcp in shell scriptCommand line srmcp in shell script FTS controlled transfersFTS controlled transfers
AchievedAchieved Peak of 948MbpsPeak of 948Mbps Transferred:Transferred:
8TB in 24 hours - 800+ Mbps aggregate rate8TB in 24 hours - 800+ Mbps aggregate rate 36TB in 1 week - 500+ Mbps aggregate rate 36TB in 1 week - 500+ Mbps aggregate rate
Over 800Mbps when running, but 0Mbps in downtimes a Over 800Mbps when running, but 0Mbps in downtimes a problemproblem
Parallel file transfers increase rateParallel file transfers increase rate Better utilisation of bandwidthBetter utilisation of bandwidth Staggered initialisation of transfers reduces overhead from Staggered initialisation of transfers reduces overhead from
initialisation/cessation of individual transfers. Rate increase initialisation/cessation of individual transfers. Rate increase from 150Mbps to 900Mbpsfrom 150Mbps to 900Mbps
2% (18Mbps) reverse traffic flow for 900Mbps transfer2% (18Mbps) reverse traffic flow for 900Mbps transfer
FTS transfers not yet as successful as srmcp only transfersFTS transfers not yet as successful as srmcp only transfers Greater overheads?Greater overheads?
More optimisation needed More optimisation needed Single FTS file transfer gives 150Mbps Single FTS file transfer gives 150Mbps
Same as srmcp Same as srmcp Concurrent FTS file transfers scales at lower rate than srmcpConcurrent FTS file transfers scales at lower rate than srmcp
All single stream transfersAll single stream transfers FTS tests currently used single source fileFTS tests currently used single source file
Srmcp used with multiple source filesSrmcp used with multiple source files Rate varies dependent on directionRate varies dependent on direction
Possibly explained by difference in dCache setupPossibly explained by difference in dCache setup V0 DependencyV0 Dependency kernel settingskernel settings Disk I/O limitationsDisk I/O limitations SRM pool load balancingSRM pool load balancing To be investigatedTo be investigated
File size affects rate of transfer File size affects rate of transfer Single stream rate varies 150 to 180 Mbps with increase from 1 to 10 GB file sizeSingle stream rate varies 150 to 180 Mbps with increase from 1 to 10 GB file size
Lancaster<->SARA LinkLancaster<->SARA Link
Link not yet activeLink not yet active Tests similar to Lancaster-RAL and Lancaster-Edinburgh Tests similar to Lancaster-RAL and Lancaster-Edinburgh
Tests Tests Bulk File TransfersBulk File Transfers UDT Protocol TestingUDT Protocol Testing
Study of effect of International/Extended link lengthStudy of effect of International/Extended link length SARA storage capacity underused, RAL capacity SARA storage capacity underused, RAL capacity
currently too small for UK simulation storagecurrently too small for UK simulation storage Also, SARA to test ATLAS Tier1 fallback scenario (FTS Also, SARA to test ATLAS Tier1 fallback scenario (FTS
catalogues etc.)catalogues etc.) Are we capable of connecting to an alternate Tier1?Are we capable of connecting to an alternate Tier1?
Lancaster<->Manchester LinkLancaster<->Manchester Link
Intra-Tier2 site TestingIntra-Tier2 site Testing ““Homogeneous Distributed Tier2”Homogeneous Distributed Tier2” dCache Head node at Lancaster, pool nodes dCache Head node at Lancaster, pool nodes
at both Lancaster and Manchesterat both Lancaster and Manchester Test Transfers to/from RALTest Transfers to/from RAL Test of Job submission to close CE/WN’s Test of Job submission to close CE/WN’s Possible testing of xrootd within dCachePossible testing of xrootd within dCache
www.eslea.uklight.ac.ukwww.eslea.uklight.ac.uk Connecting to UKLightConnecting to UKLight DocumentsDocuments