While security vendors will sell you wares for any logical layer and all possible locations on your corporate network, IP Security-based VPN gateways have emerged as the most popular class of product for setting up secure, site-to-site connections.
In our evaluation of 13 products in this market, we looked at the standard manageability, performance and enterprise-focused feature criteria, but we also added a new series of interoperability tests to the mix.
In all, it was an astonishingly close race. No one product stood out as the winner across all categories. This give-and-take is reflected in our scorecard, where more than half the products are within a point of each other in the final tally. Due to the extremely close scoring, we will not be awarding a Blue Ribbon Award for this test.
Because of ambiguities in the IPSec standard specification, not every vendor’s product will work with every other vendor’s gear – even though each might have a “correct” protocol implementation. That makes interoperability one of the greatest challenges for VPN vendors.
We set up a hypothetical security policy for a large, multisite network and evaluated how well each VPN product could fit into that network. With multiple data centres and branch offices with switches, routers and firewalls, our test bed was designed to resemble a standard enterprise data services network. We tested interoperability of each product against every other VPN product, both in setting up initial secure connections and in maintaining long-term operation over a matter of days. Specifically, we rated how each product worked with the others, worked with our certificate authority and with popular VPN client software, and how well each handled different VPN authentication methods.
An important part of our evaluation was making products work in a full mesh. While almost every VPN vendor can do bake-off style interoperability (where they successfully negotiate a security association with one other vendor), we wanted to see what network professionals would be faced with trying to pin together a true multivendor network securely.
Our top interoperability category rating went to Secure Computing’s Sidewinder. Right behind Sidewinder were products from Avaya Inc., Check Point Software Technologies Inc., Cisco Systems Inc., Microsoft Corp., Nortel Networks Corp. and Nokia Corp.
Although the objective scores based on our predefined evaluation criteria put Cisco almost at the top, we found severe restrictions in its implementation. Cisco has been both saddled and blessed with its command-line configuration interface. While some network managers love it, VPN designers will find it less than comforting. To keep VPN configuration manageable within the command line environment, Cisco allows for only a single Internet Key Exchange (IKE) policy per system. While that worked fine for our test (we wanted a single corporate policy), it wouldn’t work well in an environment where IKE policy varied across sites. In fact, when we added the SafeNet client into our interoperability test, we were forced to break our Cisco PIX configuration: it could work with the policy we selected, or with the SafeNet client, but not both at the same time.
We used digital certificates issued by an Entrust public-key infrastructure (PKI) server for site-to-site authentication and discovered a huge variability in how the VPNs under test interacted with this PKI. Unauthorized access to company networks is a big problem, and one that VPNs can help to solve.
Sidewinder, Nortel’s Contivity 1600 and Check Point’s VPN-1 all provided decent evidence that we were in control of who was coming into our network and what resources they had access to. All three products let us specify in as little or as much detail exactly what kinds of certificates were needed to get into our network. In contrast, Lucent’s Access Point 1000 and Cisco’s PIX 525 and IOS (Cisco entered both IOS on an optimized 7140 VPN Router platform with hardware acceleration and PIX 525 as site-to-site products with different capabilities and design goals) had a more relaxed view of access control: Once you got a certificate from the PKI, you were free to make any kind of VPN connection you wanted. This could be a killer in an extranet environment, where not everyone is sharing the same PKI and where you got your digital certificate is often just as important as what the certificate says.
Hewlett-Packard’s VPN Server Appliance, RapidStream’s 2000, RedCreek’s Ravlin 7160 and WatchGuard’s FireBox III 4500 all failed our certificate interoperability test. In the case of RedCreek and WatchGuard, neither adequately supports external certificates for site-to-site authentication.
Microsoft’s Windows 2000 Server with its built-in VPN service and Lucent’s Access Point supported certificates, but had trouble talking to other certificate-supporting products. For example, with Lucent’s Access Point, we had to drop back to preshared secrets (a more compatible but less desirable authentication system) when talking to Avaya and Nokia. Additionally, the Lucent product failed when talking to Cisco’s PIX with our desired policy.
HP also had problems related to strange configuration requirements in its IPSec implementation. HP either wants to always initiate the IPSec security association, or never initiate the security association. In site-to-site networks, this is rarely the way things work. Thus, we had a hard time maintaining HP security associations, because of collisions when changing keys.
WatchGuard and RapidStream also had poor interoperability results. In WatchGuard’s case, its unchangeable default security policy was insecure because it uses Data Encryption Standard (instead of Triple-DES) encryption and Diffie-Hellman Group 1, so we had to make a special case configuration in all other products to match the WatchGuard inflexibility. RapidStream 2000 gave us inconsistent results, sometimes accepting security associations and then failing to pass traffic, sometimes working perfectly.
Enterprise Manageability
As VPNs move out of pilot mode into implementation, configuration and management of dozens or even hundreds of VPN devices becomes a major issue. The only three products we tested that have GUIs fit for multisite, multivendor VPN management were Avaya, Check Point and Nokia. We took into account how these tools managed vendor-specific gear and how well each assisted in our interoperability scenarios.
Avaya and Nokia rose to the top of the enterprise configuration heap quickly. Both management tools use simple building blocks to define VPN network topology.
We created, modified and managed multivendor VPN configurations easily with these tools. Nokia’s VPN Policy Manager GUI lets you build complex topologies: combinations of hub-and-spoke; mesh; links between the two; and individual tunnels between any two systems.
Avaya’s GUI offers less topological power, but has a stronger back end. Built on top of a Lightweight Directory Access Protocol server, this product lets multiple network managers view, edit and push changes to a large network from multiple sites. Avaya’s VPN Manager GUI also has a built-in client deployment and management tool for remote access VPNs. Nortel’s Contivity 1600 has a similarly powerful tool built in, but the scope does not include more than one VPN security gateway.
We evaluated Check Point’s recently released version of VPN-1 and were pleased to see that the company has come a long way in terms of VPN configurability and management since we looked at its V4.1 GUI earlier this year. Check Point’s Policy Editor GUI can now handle a larger range of interoperability scenarios. Although we tracked down a few bugs in the new software, our general impression is that Check Point is focusing more on large VPNs. The Check Point GUI can easily generate meshed VPN topologies – a significant improvement from the V4.1 GUI, which tightly restricted a VPN manager’s flexibility in selecting operating parameters.
Check Point’s management tool also allows integration of firewall rules and VPN settings. Although Check Point’s product does not support the comprehensive array of protocols and features that Lucent’s Access Point does – which lets you stack firewalls and VPN tunnels in any configuration you’d ever desire – it handles 90 per cent or more of the configurations most enterprise managers would dream up. The ability to express firewall filtering and VPN tunnelling in the same rule is an essential element in merging VPN and firewall policy.
Several of the VPN management tools were not very useful in our interoperability quest. Nortel’s famed Optivity tool is good for doing things to multiple Contivity boxes without having to touch each one, but doesn’t really do anything for site-to-site VPN configuration, even where only Nortel systems are included. This is unfortunate because Nortel’s element management (via a local Web server on the Contivity system) is the best system management tool of any product we evaluated. If Nortel could extend that management to multiple systems, it would have the knock- down winner in this category.
Lucent’s optional QVPN Builder GUI went the distance in terms of managing a network of VPN devices as a single unit, but had a critical flaw: It was not designed to have anything but Lucent Access Points in the network. Because of this, we fell back to Lucent’s command-line interface (CLI) for configuration. Unfortunately you can’t use both the QVPN Builder and the CLI; you have to choose one. HP’s management tool for its VPN Server Appliance has a similar restriction: It has some primitive capabilities to apply a single policy to multiple systems but does not have any way to integrate non-HP products into the configuration. WatchGuard’s optional VPN Manager software leaves network managers in much the same boat. The lack of firewall integration of the VPN configuration also left us disappointed because WatchGuard’s firewall configuration is so simple and intuitive.
Microsoft’s VPN management is far from simple and intuitive. Although it is possible to create a single IPSec policy that could be applied to multiple systems – as long as they are all Win 2000 servers – the GUI is complex and confusing. After 92 screens, we still couldn’t figure out whether it was better than the competition. Frankly, any security configuration tool this complex is asking for an error to happen, which is unacceptable in an enterprise network.
Cisco’s new Cisco Secure Policy Manager will be a great boon to any network manager interested in using either Cisco PIX or its IOS-based systems as a firewall or VPN. With Cisco’s CLI syntax slightly different from and incompatible with IOS to PIX, Cisco Secure Policy Manager wins major points for making it possible to design and synchronize firewall rules and intrusion-detection systems across multiple Cisco systems. In this case, though, “possible” doesn’t mean “optimal”.
Cisco Secure Policy Manager has dedicated tools for building VPN tunnels. Both mesh and hub-and-spoke topologies are supported. Unfortunately, there is no support for third-party VPN products – you have to fake out Cisco Secure Policy Manager by describing them as Cisco elements. More importantly, the VPN configuration is not well integrated with the firewall rules.
Other Enterprise Features
No VPN exists in a vacuum. VPN functions may need to be combined with other parts of the enterprise network. Common additions to VPN devices on the market include firewalls, high-availability features, routing protocol support, bandwidth management and quality of service (QoS), multiple interface options (besides Fast Ethernet, which is most common), and tunnel status monitoring and reporting. Technically, these are not VPN-specific features, but enterprise managers will find them useful adjuncts in their quest to build more powerful and capable networks.
One obvious winner in the “how many features can we add to a single box” game is Cisco. Its IOS-based product includes VPN capabilities as a sideline, with industry-leading routing, multiple interfaces, high availability, traffic shaping and firewall all built into the same system.
Microsoft’s Win 2000 Server could also be considered the feature king – what other VPN device can also run PowerPoint and Flight Simulator? While more relevant features, such as simple routing protocols, traffic prioritization and basic firewall, are easily available, Win 2000 Server has an advantage when it comes to database operations for remote users. Because the Win 2000 VPN software is completely integrated with the Windows Active Directory authentication system, access control for remote users can be tightly controlled and managed from any Win 2000 system.
The obvious combination of VPN and firewall features makes collocation more the rule than the exception, with virtually every product we reviewed having at least limited firewall capabilities. The two holdouts are Nokia and RedCreek. (RedCreek announced an integrated firewall in its VPN product as this review was going to press.) We found Check Point and Secure Computing’s approaches the easiest to deal with. Both offer an integrated firewall and VPN rule set. WatchGuard and HP are examples of the other extreme: The firewall and VPN are totally separate and disconnected services, which happen to share a GUI and sit on the same system.
Routing is another area where integration between firewall and VPN is important. Products like Nokia’s CryptoCluster and Check Point’s VPN-1 have no real internal connection between routing protocols and VPN status – tunnels come up, tunnels go down and the routing algorithm is none the wiser for it. Lucent’s Access Point does an excellent job of integrating the two, representing tunnels as interfaces, which lets the routing system propagate information about the state of the VPN. This can be critical when a VPN tunnel is part of a back-up strategy or when multiple redundant paths exist across a VPN. In recent versions, Cisco’s VPN products have added similar integration.
High-availability functions varied between products. Nokia, the unchallenged leader of the high-availability load sharing cluster, focuses on reliability at a single point in the network. Other vendors, including Avaya and Nortel, built in reliability from a multisite or multilocation point of view.
Service-level monitoring and reporting can be important in secure enterprise networks. Lucent and Check Point have built-in tools to monitor the latency and loss rate of VPN tunnels and maintain an internal database of performance statistics that can be used for long-term charting or alerting purposes.
QoS is another enterprise-level feature that varies among products. For example, several of the VPN security gateways we evaluated have the ability to mark Differentiated Service bits on packets. Avaya’s VSU series does this, but won’t change its packet handling options based on QoS markings. This is primarily useful where other components in the network handle bandwidth management. Other products, including Nortel’s Contivity and Cisco’s IOS, mark and allocate bandwidth to tunnels based on configured-in rules.
Performance
We’ve found – both in this and in past tests – that performance of VPN devices varies widely. In many cases, vendors purposefully understate performance to drive sales to more expensive devices; in other cases, they overstate performance to make their products appear more competitive. While we did not conduct a comprehensive suite of performance tests – as that was not our primary objective for this review – we did take the opportunity to run some quick benchmarks to offer apples-to-apples comparisons of these products.
We ran three sets of performance numbers, evaluating behaviour in best-case and worst-case packet flows, as well as with a typical Internet mix. For the Internet mix, we used data collected from an Internet backbone to build a profile of approximately 50 per cent small packets (96 octets or less), 10 per cent large packets (1,518 octets, the Ethernet maximum transmission unit), 20 per cent 576 octets (a common WAN MTU) and 20 per cent assorted between 192 and 1,024 octets.
We discovered that for line speeds of up to 10Mbps (full duplex, about a quarter of a DS-3/T-3 circuit), any of the products can keep up – but Avaya, Nortel, RapidStream and Microsoft give you excellent price/performance ratios.
If you want to push to a full DS-3 circuit (45Mbps, full duplex), again using “real world” packet sizes, only Lucent’s Access Point with dual cryptographic accelerators and the one-two punch of Win 2000 combined with Intel’s Pro/100S cryptographic network interface cards (NIC) beat the 90Mbps needed to handle that circuit. By adding less than US$200 worth of hardware to our system, we drove total IPSec performance of Win 2000 up to more than 160Mbps in the best case (large packets). Given the low cost of Pentium-based PCs, Win 2000 Server software and the Intel NICs, this particular packaging achieved price/performance ratios between 10 and 20 times better than the other vendors’. However, we note that our performance tests were done with only six IPSec security associations. As a central site system with 500 security associations, we saw total performance of our Win 2000 system drop dramatically to less than 8Mbps for the Internet mix.
Nokia offers a load-sharing product and we tested it in two ways: as a single stand-alone system (a single CryptoCluster 2500) and as a cluster (three clustered CryptoCluster 2500s). The results were fairly dramatic, showing almost linear growth in performance along with growth in cluster size.
Breadth of Product Line
Enterprise network managers often need to mix everything from dial-up modems to 155Mbps OC-3 lines in the same network. For that reason, a one-size-fits-all approach won’t work in the real world. This is one reason multivendor interoperability is so important: Small-office/home-office (SOHO)-sized products from vendors such as RedCreek and WatchGuard may be the right fit for some parts, perhaps talking back to gigabit behemoths like the Nokia 5205.
At the same time, a single vendor immensely simplifies management, as we saw so dramatically, and a broad product line can be an important advantage. Therefore we also looked at the breadth of VPN product line in our rating equation. We wanted to know how well vendors’ product lines reached up to data-centre-sized encryption engines and stretched down to the SOHO market with inexpensive simple devices. We also evaluated how well network media other than the vanilla Fast Ethernet were included: WAN interfaces such as T-1/T-3 and ISDN can be important in keeping costs down, while Gigabit Ethernet is critical for the high-end data centre.
The obvious winner in this area is Cisco’s IOS. With more than a dozen chassis options, products ranging from less than US$1,000 to the Internet-core-sized GSR 12000, with its six-figure price tag, and interfaces ranging from built-in modem up to Gigabit Ethernet – no one beats Cisco’s IOS when it comes to breadth of product line.
However, high scores in this category don’t necessarily equate to the same level of flexibility, so investigate what hardware and speeds you need in this area carefully. For example, Cisco’s PIX 525 rated well because of the large number of simultaneous interfaces supported, while Nokia’s CryptoCluster 2500 got the same score because they stretch from branch office to data centre in speed – although Nokia only supports two interfaces on its systems.
This review shows there is room in the VPN marketplace for more than a handful of vendors. Each product evaluated has specific strengths and weaknesses; each is designed with a different kind of network, management style, VPN size and set of requirements in mind.
Client Interoperability
Generally, remote access client interoperability is not a big concern. Most companies have only a small number of remote access gateways and will typically select a single vendor to provide the gateways for their users. The client software provided by the gateway vendor is then the obvious choice for the company.
Two companies have made a business of providing interoperable clients for IP Security (IPSec) access to enterprise networks. Safenet (formerly IRE) sells its Windows-only client in an OEM configuration to many VPN vendors. Network Associates’ PGP division offers Windows and Macintosh versions of an IPSec client application. We invited Safenet to come in for a brief interoperability test to see how well things would work.
Results were mixed. In the best case, Nokia and Check Point Software gateways take Safenet as a completely compatible client right out of the starting gate. Some gateways, including those from Nortel, Lucent, Cisco and Secure Computing, required that we make minor changes to our profiles before they would handle Safenet clients properly. Other gateways, such as those from Avaya, Microsoft and Hewlett-Packard, were unfriendly to foreign clients. They wouldn’t work unless the client looked like a gateway with a static IP address, which is not generally useful for remote access client configurations. Finally, other vendors, including Cisco with its PIX entry, the configuration of the system was so restricted that by adding client compatibility we broke site-to-site interoperability.
How We Did It
We used equipment from Cubix and Spirent Communications to build a test bed surrounding the VPN security gateways under test. We used the Cubix Density system running custom software to create VPN tunnels, verify connectivity between the gateways, measure up-time and display the full connectivity matrix.
We developed our test methodology with engineers from the VPN Consortium and will present a demonstration of this interoperability test at the upcoming VPNCON trade show in Alexandria, Va., from Oct. 15 to 18.
We created an IP Security (IPSec) profile, which we believe a prudent network manager would want for a corporate VPN network.
This profile included Internet Key Exchange encryption algorithm of Triple-Data Encryption Standard and authentication of secure hash algorithm 1 (SHA-1) using Diffie-Hellman Group 2 (MODP-1024) and a lifetime of 8 hours; IPSec encryption algorithm was Triple-DES, authentication was SHA-1, perfect forward secrecy was enabled for Diffie-Hellman Group 2, and lifetime was one hour.
Our test bed comprised multiple data centres and branch offices with switches, routers and firewalls. We tested the interoperability of each product against every other VPN product both in setting up initial VPN connections and in maintaining long-term operation over a few days. Specifically, we rated how each product worked with each other product, with our certificate authority and popular VPN client software, and how well different VPN authentication methods worked in our hypothetical network.
To handle authentication of the security association, we used an Entrust public-key infrastructure (PKI) to pass out digital certificates to each security gateway. Because Entrust’s PKI corners a significant share of the market, it was reasonable to expect all devices to support it, at least in manual enrolment mode. Some devices supported simple certificate enrolment protocol enrolment to our Entrust PKI; with others, we used manual enrolment to the Entrust server. We used preshared secrets for authentication in cases where the security gateway did not support digital certificates (or our Entrust PKI).
To capture performance for these devices, we used a set of six Nokia CryptoCluster 2500 gateways against each system being tested. This configuration is sufficient to saturate a 100Mbps full-duplex Ethernet network with 64-octet packets. We generated User Datagram Protocol (UDP) packets of various sizes using Smartbits gear and off-the-shelf test software, and measured when loss went above 0.1 per cent using a precision of 2Mbps.
Joel Snyder is a senior partner at Opus One in Tucson, Ariz. He can be reached atJoel.Snyder@opus1.com.
