Until now, storage-area networks have mostly been local networks, providing high-bandwidth, low-latency interconnect for the high-speed movement of block data from server to storage.
But, if all the data that needs protection is under one roof, it is vulnerable to natural disasters such as fires, floods and earthquakes. What is needed is a method of interconnecting multiple SAN islands over distance.
SAN extensions must provide high performance and work within today’s datacom/telecom infrastructure. Fibre Channel over SONET provides such a solution. The protocol and gateways introduce low latency into the system.
Imagine a primary data centre and a mirrored site each with hard disks operating at 15,000 rpm and 2 msec seek time. If data is written to both disks, we will first get a response from the local disk based on the 2 msec average seek time.
However, the response from the mirrored site will be delayed by the round-trip time to send the data. This round-trip time has two components: the length of the fibre between the two installations and the latency of the network equipment used to connect the Fibre Channel SAN to the fibre.
The latency of an optical signal travelling in a fibre-optic cable is 5 microsec per kilometre. If we locate our mirror site 200 kilometres away, the fibre alone will add 2 msec of delay to the response of the remote disk.
Now, the average latency for the remote SAN has been increased to 4 msec. Because the remote SAN is a mirror to the primary data centre, the overall latency of the system has been doubled. We can think of our state-of-the-art 15,000 rpm, 2 msec seek-time disks as now being 10,000 rpm, 4 msec seek time drives. All the bandwidth in the world won’t make this go away. Latency is a separate phenomenon from bandwidth. Both affect the throughput of a system but in different ways.
Unfortunately, the fibre-optic cable is not the only component of latency. The Fibre Channel gateway will introduce latency as Fibre Channel is encapsulated for transport over IP, ATM, SONET or dense wavelength division multiplexing (DWDM).
Depending on the architecture, a gateway may have multiple internal stages. Each stage introduces queuing delays and other forms of latency. A large, multiprotocol gateway might introduce hundreds of microseconds or even milliseconds of latency, whereas a highly integrated chip-based solution might introduce only 100 microsec or less of latency.
What matters is the total latency from end to end. Fibre length and gateway equipment contribute to that latency. Both should be evaluated as the storage network system is planned.
Why SONET?
Of the methods mentioned, Fibre Channel over IP provides scalability and convenience while Fibre Channel over ATM can be readily extended over distance (albeit with reduced data rates of 155Mbps or 622Mbps). Unfortunately, both techniques can introduce substantial latency (up to tens of milliseconds).
On the other hand, SONET and DWDM can provide low latency, high-bandwidth transportation of Fibre Channel traffic. The problem with DWDM is that the equipment is expensive and requires dedicated fibre to interconnect the sites. Dedicated fibre that runs 200 km is prohibitively expensive.
Fibre Channel over SONET emerges as an excellent choice for carrying Fibre Channel data. It is readily available from many carriers at rates up to 2.488Gbps and is already part of their existing infrastructure. Fibre Channel over SONET gateways can be built to introduce only minor amounts of latency (approximately 100 msec or less).
Because Fibre Channel over SONET does not use routers that drop packets under congestion, TCP and its attendant retransmission delays are eliminated. Also, Fibre Channel over SONET can be readily extended hundreds or even thousands of kilometres without installing dark fibre.
Andy Helland is the director of product management at LightSand Communications Corp. He is also one of the authors of the Internet Engineering Task Force standard for carrying Fibre Channel over IP. He can be reached atandyh@lightsand.com.
