My first reaction was not exactly unbridled cooperation. I knew right away that redundancy would have to be removed and that old habits such as private circuits and the latest networking equipment were now in the past. I could already hear the users complaining about robotic-sounding phone calls and choppy videoconferences. Worse, the people who ask you to cheapen things are usually the first to complain when there’s a problem.
In the end, we fell short of the 40 per cent target, but we came close. By moving to 100Mbps Ethernet, we’re conserving about US$40,000 per year in WAN charges — a 33 per cent savings compared to the cost of our private T1 circuits.
WANs yesterday and today
From a 30,000-foot view, a WAN needs to be 100 per cent reliable and available 24/7. In a perfect world, this would be done with redundant, private data circuits in each office location. Redundancy is not just a matter of backup circuits and redundant routers and switches, but also leveraging multiple vendors; if one vendor has an issue, then the other vendor (hopefully) would still be chugging along.
Other times, you may look at using a cheap Internet service provider with your own point-to-point VPN tunnels to get data between your different locations. Be warned: If you decide to use a cheap Internet line between your different sites, you will be plagued by issues such as high latency and inconsistent data speeds — especially with unforgiving traffic such as phone conversations and videoconferencing.
As happens with many businesses, our WAN had changed and evolved as our company grew. At one point, every location was connected by private T1s back to headquarters. As more bandwidth was needed, more and more T1s were thrown at the problem. Several years ago, we decided to move to a corporatewide MPLS network that would allow the different locations to talk directly to one another; HQ would no longer serve as the central hub. The MPLS pipes into the different locations were scaled depending on the size of each office, with HQ getting the lion’s share of bandwidth because a majority of the traffic still flowed there.
If only replacing the expensive WAN were as simple as picking a new vendor and having new circuits installed. Unfortunately, there is a lot more to it. You need to wade through an ocean of contracts and terms, find which vendors serve all of your locations, and be ready to navigate through salespeople promising the sun, moon, and stars, only to discover their companies won’t deliver a fraction of what they promised. You must also scrutinize existing contracts with service providers to make sure you can break with them and determine whether there is a penalty for doing so.
You’ll also need to take a close look at your traffic statistics and your traffic flow. Every good network engineer does monitoring not just to track uptime of systems but also network utilization. Your network is the circulatory system of your business. As you look at the statistics over time, you will see patterns to the data flows that will depend on the distribution of different types of users across your locations. For example, one site might experience numerous videoconferencing calls back to HQ because satellite teams or marketing people work there. Another site might have mostly one-way data traffic because the users there are just downloading emails from the mail server at headquarters. In our case, we saw the majority of our traffic flowing between our two biggest sites, so these were given the biggest pipes.
Your own private Internet
Ultimately we decided on 100 Mbps pipes for HQ and our second largest office, both of which could be served by the same provider. These circuits are sold as large Internet pipes. Nevertheless, by using VPN tunneling and “skating” across the provider’s internal network, we found that network latency was slightly better than what we had with our dedicated MPLS network.
Our smaller offices received much smaller Internet connections via local ISPs, making use of the same type of VPN tunneling. However, the local ISP that serves the small offices does not have guaranteed QoS, so the videoconferences to those locales became spotty. Using our network logging statistics, we were able to convince the office managers to give up videoconferencing altogether. We could show that videoconferences were rare, and that it made no sense to pay a premium for a data pipe that would scarcely be used.
Users generally don’t care what type of WAN infrastructure you have in place, but they do expect the WAN to work all the time and every time. To achieve the cost reduction we needed, redundancy had to go. This was the hardest piece to give up, and sure enough, it has come back to bite us. We have at least one or two short outages every month. Typically these last an hour or so. During an outage, all communication stops. There is no inter-office calling (users have to dial long distance), no videoconferencing, no email, and no Internet.
In our yearly satisfaction survey, these outages were noted by numerous people including most of the senior executives, who realized the cuts may have gone a little too far. As a result, we’ll be getting cheap cable modem lines for the two big offices. We won’t be able to go back to 100 percent redundancy in all locations, but we are at least bringing some redundancy to our larger sites.
Abandoning private WAN lines for fast Internet connections and VPNs isn’t for everyone. The old way is proven and reliable, and I encourage you to stick with it if you can. But in this economy, everyone has to make cuts. If your WAN falls under the budget axe, know that you can build something similar on the cheap. Just be sure to insist on some kind of redundancy, even if it is only a DSL or cable modem.