Editor’s Note: This article was researched and published in Network World (U.S.) prior to the devastation that occurred Sept. 11 in Manhattan’s financial district. The current status of Lehman Brothers’ metropolitan-area network is unknown.
As any real estate agent will tell you, it is better to own than rent. The same can hold true for metropolitan-area networks.
While many service providers now offer high-speed Ethernet MANs, some corporations – both large and small – are taking the “do-it-yourself” approach. Despite challenges – which include cutting through government red tape to lay fibre, and integrating carrier-class optical routing into an enterprise network – users are finding that virtually unlimited bandwidth for no monthly charge is worth it.
One city where large-scale private MANs are under construction is New York – the quintessential “metropolitan area.”
Sheng Guo, CTO of the State of New York court system, is overseeing the installation of a private, seven-site dense wavelength division multiplexing (DWDM) network connecting courthouses in Manhattan and the surrounding boroughs, as well as Westchester and Nassau counties.
The New York City-area DWDM network is the next phase in CourtNet, the state’s OC-3 (155Mbps) speed network installed in some Manhattan and upstate New York courts and jails. The system lets judicial hearings be held by videoconferencing, allowing prisoners incarcerated in one part of the state to appear at court hearings hundreds of miles away.
Although the project’s total cost is estimated at US$50 million, “CourtNet has a huge payoff,” Guo says. “It’s really changing how people interact. Just a few years ago, no one even had e-mail. Now we can do video face to face.”
Guo says videoconferencing has saved the courts on the costs of transporting prisoners to hearings across the state, which can cost hundreds of dollars for a single trip, when travel expenses and overtime pay for prison guards is factored in.
The DWDM network will be built on a 140-mile fibre-optic ring, and will connect 8,000 court employees in the New York City area and allow the CourtNet video service to be expanded to courthouse campuses in Brooklyn, the Bronx, two locations in Queens, and Nassau and Westchester counties.
An Optera DWDM switch in the court system’s Manhattan data centre will switch 14 channels, or wavelengths of light, on the ring, providing 2Gbps to 3Gbps of bandwidth per campus.
Optera switches attached to the ring at each campus will only transmit and receive their respective channels, allowing other sites’ channel traffic on the ring to pass through. Each site will get two to three dedicated channels of light, with each channel representing a 1Gbps dedicated link back to the Optera switch in Manhattan. The court system is working with Metromedia Fiber Network, a New York fibre-optics provider, to install the dark fibre ring.
Guo says this network will also be the foundation for the court system’s next project, which is to put the court’s multivendor PBX phone system on one IP network.
The motivation for the DWDM network was to provision bandwidth in-house, rather than rely on a service provider. Guo looked at some MAN providers that offered metropolitan Ethernet connectivity, but they didn’t have the speeds he needed.
“Since we have the dark fibres and we’re using our own equipment, the control of the bandwidth is totally up to us,” Guo says. “If you have the dark fibre, it doesn’t matter how much bandwidth you put through it.”
Financially Sound
Across town at Lehman Brothers, the financial services firm is also using DWDM gear, as well as a mix of Gigabit Ethernet and traditional WAN gear, to connect its offices and data centres in Manhattan and Jersey City, N.J., which support 8,000 employees in the area.
Lehman Brothers’ MAN connects five buildings located throughout downtown Manhattan, which are linked with a combination of T-3 lines and Gigabit Ethernet links over leased fibre.
These office networks tie into a data centre located at the World Financial Center. Four Cisco ONS 15000 optical transport system boxes – a pair on each side of the Hudson River – are used to bridge the Jersey City and Manhattan data centres.
“Using DWDM gives us much more bandwidth than a leased line ever could,” says John Manville, senior vice-president of network services at Lehman Brothers. “It also enables us to use a few dark fibres much more cost efficiently than buying more and more dark fibre circuits from a carrier,” which is still relatively expensive.
The firm’s backbone supports myriad traffic, including Web and database applications, financial modelling and other corporate applications in Lehman’s data centres, as well as IP voice traffic for the 1,000 Cisco IP phones and IP PBXs Lehman has deployed in its offices.
EMC storage arrays in both data centres store the firm’s trove of financial data and provide back-up for each other in case of an equipment failure in one of the two centres.
To handle all this, the ONS 15000s multiplex 10Gbps of traffic onto a channel, or wavelength of light, on a single-fibre pair. Lehman’s backbone utilization runs around 20 per cent to 30 per cent (or 2Gbps to 3Gbps), Manville says, and never exceeds 50 per cent, even during peak periods, such as data back-ups between EMC storage systems.
The advantage of a private DWDM, Manville says, is that more bandwidth can be added by multiplexing more ports over more channels, to a maximum of 32.
IP routing and link failover are handled by Cisco Catalyst switches that feed into the ONS boxes. While the DWDM technology sounds complex, Manville adds, “What we have now is actually pretty simple….The [DWDM] boxes don’t have many features on them.” The basic task the optical routers perform is the shuttling of traffic from Point A to Point B.
Much of Lehman’s metropolitan network in Manhattan consists of Gigabit Ethernet links connected with more than 150 Cisco Catalyst 6509 switches.
Manville says the company could have connected the New Jersey data centre into the Manhattan network by plugging the fibre connections into two Catalysts on opposite sides of the river. But using straight Gigabit Ethernet without DWDM would have been comparable to building the George Washington Bridge with only two lanes of traffic.
“We could have stuck a dark fibre pair between two 6509s and got one Gigabit between them, and it would have worked fine,” Manville says. “But that wouldn’t have addressed the EMC [bandwidth] requirements, and it wouldn’t have addressed very well the redundancy requirements, if at all. If the dark fibre was very cheap, maybe it would be a different matter.”
Beyond The Big City
You don’t have to be a global financial institution or a large government agency to string together your own high-speed Gigabit MAN. And you don’t have to be in a big city either. Just ask Ted Malos, director of technology for the Ventura Unified School District in Ventura, Calif. With the help of the local cable company, Malos is in the process of wiring together 25 schools and other facilities throughout the district with fibre-based Gigabit Ethernet links.
“We plan to eliminate all our Cisco routers and connect metro fibre right into our switches at each site,” he says. “We’ll be going Gigabit between the sites, but using off-the-shelf equipment.”
By doing this, Malos says, the school district will increase its MAN bandwidth more than 600 times, while eliminating the monthly charges for T-1 circuits, which cost the school US$125,000.
Instead of carrier-class optical gear, the technology used to connect the campuses together will consist of stackable Layer 3 switches from 3Com, specifically the SuperStack III NBX 3300 and 4400, which can also switch at Layer 4. While it’s not the most advanced technology, the 3Com equipment “will give us the packet routing features we need to keep campus traffic flowing,” he says.
The network now has Cisco routers linking the school district’s campuses together over T-1 lines. The opportunity to put in a Gigabit Ethernet MAN for the school district came up when the city renegotiated its deal with local cable providers.
As part of the agreement, the cable company offered to install the fibre-optic cable for the school district at cost. The fibre is owned by the school district, which pays the cable company
With the school district’s WAN bandwidth going from 1.54Mbps to 1000Mbps, Malos says a whole new set of applications, such as IP video and voice, are now a possibility. A 3Com NBX IP phone system has already been ordered by the school district, and will go live districtwide by November, he says.
