In a project that could enable bigger SANs and computing grids, nuclear physicists have built a native 10 Gigabit Ethernet network linking Switzerland and Japan. The 18,500 kilometer LAN crosses 17 time zones, as it follows fibres through Amsterdam, Chicago and Seattle.
Tokyo University’s GRAPE-DR project is using data reservoirs at each end to optimize large TCP data flows across a long fat pipe. They aim to get standard Ethernet working end-to-end, without the need to convert to other protocols for the WAN section or drop to lower speeds.
So far, the network has achieved unprecedented speeds of 9Gbps in testing between groups of Xeon servers, and 7Gbps for a single TCP packet stream between two servers equipped with Gigabit TCP offload adapters.
High speed networks are needed so physicists around the world can work on data generated by projects such as the upcoming Large Hadron Collider at CERN in Switzerland. As they build them, network engineers are learning how to configure standard TCP to run better over long distances.
“Currently, long-distance very high-bandwidth communication is very expensive and hard to fully utilize. If we solve these two problems, how to use high-speed network will drastically change in both business and technology,” says Professor Kei Hiraki of Tokyo University.
The researchers used 10Gig Ethernet WAN PHY technology to extend a LAN in Tokyo to include computers at CERN, allowing a Japanese data analysis centre to work on CERN-generated data.
“It is exactly similar to LAN except for latency from user point of view,” says Professor Hiraki. “The importance of WAN PHY technology is mainly cost for equipment at end nodes and intermediate points. For example, MAC chip and optical modules as well as controlling software are much less costly.”
The main problem for 10Gig is server connection — plenty of switches support it, but it is just too fast for today’s servers to process. The solution chosen by the Tokyo team is to use TCP offload adapters from Chelsio Communications Inc., which relieve the host server from needing to process Ethernet packets.
“We’re implementing TCP in hardware,” says Chelsio’s marketing director Randy Smith. “We can provide incredible flexibility in how you configure TCP and the headers, windows size, and so on, to target the protocol to your environment. The key here is we were able to pace the traffic, so they were able to transmit a sustained 7Gbps for seven hours.”
Chelsio’s cards cost US$2000 to US$4000 each. “Per port, 10Gig is still about 10 times the cost of Gigabit,” Smith says. “But it’s Ethernet so we will drive the cost out of it fast. 10Gig will get to copper in a year or two.”
He added that the ability to run full speed Ethernet at a distance of up to seven miles is key for the development of metro Ethernet. “We will see a big move next year for one to 10 Gigabits on the research side next year,” he says. “We’ll also see it move into the enterprise, and in storage networks and NAS as well as high performance computing.”
