CANARIE - End to End Lightpaths Used to Bypass Internet Bottlenecks for High Speed Data Transfer Between CERN and Canada, Via SURFnet and StarLight

August 13, 2003

Ottawa - The Canadian ATLAS experimental group was presented with a challenging problem: how to transport 700 Gigabytes of data from an experimental test site in CERN, Geneva back to the Physics Departments of Carleton University and the University of Toronto. A recent test at the CERN laboratory provided data to calibrate a set of 4 ton, million dollar particle detectors that were constructed in Canada and the U.S. The extensive analysis of this data to be undertaken by the Canadian groups required that the data be shipped back to Canada and accessible on local computer systems. The volume of data made it impractical to do this over existing networks. To solve this problem, a unique arrangement was orchestrated between CANARIE in Canada , SURFnet in the Netherlands, CERN in Switzerland, StarLight in the United States, ORION in Ontario, Carleton University in Ottawa, and the shared facilities of TransLight. A set of optical lambdas were configured to provide an end to end lightpath to facilitate the transfer of 700 Gigabytes of data in what can be considered the first real practical use of an international end to end lightpath.

The 700 Gigabytes of test beam data from an experimental area at the European laboratory CERN to the Department of Physics at Carleton University was successfully transferred in about 6.5 hours. To transfer data of this magnitude over the normal Internet network connection through firewalls and campus networks would have taken 67 days continuously.

The end to end lightpath, a point to point optical link, between CERN and Carleton University is a realization of an emerging and core networking paradigm of CA*net 4 and other experimental infrastructure networks for enabling data and computationally intensive research. This "optical bypass" provides a fast long wide pipe for high speed data transport, a fundamental requirement for large scale scientific experiments and global scale Grids.

The end to end lightpath concept adapts the classic Internet end to end principle to circuit switched networks and allows the end user to create, spawn and cross connect lightpaths to other users on independently managed networks around the world.

Through the months of June and July, the Canadian and U.S detector components were exposed to a test beam at CERN using one of CERN's high energy proton beam facilities. The detectors are part of an energy measuring system and are referred to as the Forward Calorimeters or FCALs. The FCALs, which are an integral part of the $500 million dollar ATLAS detector being constructed for the Large Hadron Collider (LHC) at CERN, were designed and built at Carleton University, the University of Toronto and the University of Arizona. The data will be transferred from Carleton University to the University of Toronto and the University of Arizona as soon as similar lightpaths can be established to these universities. All three groups will be analysing the data to determine the performance of the detectors and establishing optimum software analysis tools to be used when the ATLAS detector begins acquiring data in 2007. Further FCAL test data will be transmitted over this facility in the coming months.

The lightpath was established with a capacity of 1 Gbps end to end between the FCAL Linux data acqusition PC at CERN in Geneva and a 3 Terabyte Linux disk server at Carleton University Physics Department in Ottawa. Due to the small time window to actually transport the data during the running experiment, there was no opportunity to tune the network and I/O performance of the servers. Tsunami, an experimental high speed network file transfer protocol developed by the Advanced Network Management Laboratory at Indiana University was used as the transfer protocol. An average rate of about 250 Mbps was obtained.

This link was made possible with the contributions and support from Ralph Michaelis and his staff at Carleton University. ORION, Ontario's Optical Regional Advanced Network (ORAN), generously loaned the use of fiber between Carleton University and CA*net 4 for this experiment. Randy Neals, Sam Mokbel and Ron Neil from ORANO were instrumental in enabling this. Damir Pobric and Thomas Tam from CANARIE coordinated the provisioning of the lightpath between CERN and Carleton University, as well as helping to isolate local high dispersion fibers, which were problematic. Dale Theoret and Verna Murray from Telecom Ottawa, and Michael MacKenzie from Groupe Telecom assisted in identifying and interconnecting fibers at the Ottawa ORION PoP and at the Ottawa CA*net 4 PoP, respectively. Jim Yuan from the Ottawa RAN and from the Ontario Ministry of Enterprise Opportunities and Innovation (MEOI) assisted by ensuring that the interconnections were made in Ottawa.

SURFnet contributed the trans-Atlantic and Amsterdam to CERN portions of the link. Erik Radius of SURFnet and the engineers at SARA in coordination with CANARIE engineers jointly constructed the end to end lightpath. Bob Dobinson and his team from the EP Division at CERN were instrumental in extending the lightpath from the CERN IXP to the experimental area at CERN. Beniamino Di Girolamo from the ATLAS collaboration coordinated the cabling and networking in the north experimental area at CERN. Olivier Martin and his networking engineers from the CERN IT Division assisted with the connectivity at the CERN Internet Exchange Point (IXP).

This experimental use of an end to end lightpath was conducted as part of the Carleton University led CA*net 4 International Grid Testbed, supported under CANARIE's Directed Research Program. The effort at CERN is partially supported through the European Union funded ESTA project. This reflects the ongoing collaboration between ESTA and CANARIE.

For more information:

Last Updated: 18 February 2010