Competing with Cisco Systems Inc., even a somewhat humbled one, is no picnic. But, having your product lab-tested against Cisco’s marketing claims is enough to put the most grizzled network professional into a state of shock. As bizarre as it sounds, this is what took place about a month ago in the labs of the newspaper formerly known as PC Week (now called eWeek).
To make things more interesting, the matchup between Cisco and the upstart was identical to one that the upstart had commissioned The Tolly Group to run – only we actually benchmarked both vendors’ products. (Lest the reader think I’m writing this column to benefit the said upstart, the company shall remain nameless.)
The “test” involved gigabit Ethernet security devices and evaluated the performance of firewalls (single device) and VPN tunnels (matched pairs).
In addition to being misleading, it really doesn’t even deserve to be called a test. Testing firewalls and VPNs is significantly more complex than “running the numbers” on a Layer 2/3 switch. Without certain key information about such a test, the results are meaningless. The eWeek story is devoid of such details and serves as a lesson for readers in how to interpret tests.
Let’s examine this key finding: “With only 750Mbps throughput across the firewall, the [Brand X] appliance comes up a little short of the gigabit throughput that Cisco advertises for its new PIX 535 firewall. . . .”
Was that half-duplex or full-duplex? If the latter, the maximum achievable throughput would be 2 Gbps, not 1 Gbps. At what packet size was this achieved? What was the packet-loss tolerance? And, most importantly, how many sessions were flowing across the firewall?
Our test shows Brand X achieves a maximum throughput of 757.44 Mbps when running full-duplex streams of 1,518-byte packets. The packet-loss tolerance is 0.001 per cent (the same we use for Layer 2/3 switches) and the test gear was simulating 25,000 sessions. This, eWeek tells us, should compare to “gigabit throughput” claimed by Cisco.
We ran this test on the Cisco PIX 535 (IOS 5.3) gear -or tried to. Using our standard packet-loss tolerance of 0.001 per cent, the highest throughput we could achieve was 1.94 Mbps. Yes, around 2 Mbps out of a possible 2 Gbps – and that was with just 1,000 sessions.
To squelch possible complaints that our test was too stringent, we lowered the bar by a factor of 1,000 and let the tests continue as long as packet loss remained below 1 per cent (an unacceptably high loss rate).
On those tests, Cisco achieved 675.29 Mbps – at the 5,000-session level. When run with 10,000 sessions, the throughput slid to 199.38 Mbps before falling off the cliff delivering 38.66 Mbps with 25,000 sessions. Brand X sailed along at approximately the 750 Mbps at all session rates.
As with all of our tests, we invited Cisco to participate in evaluating the methodology, prototype and results. As the company almost always does, it declined. Prior to publication, Cisco acknowledged a “bug” when handling more than 2,000 sessions. Every 30 seconds, the PIX stops forwarding for about four seconds. Lovely.
Good testing is difficult, and details matter. So, be careful not to fall into the trap of taking pseudo-tests as reliable input for your product-purchasing decisions.
Tolly is chairman and CEO of Tolly Research. Tolly also is founder, president and CEO of The Tolly Group. He can be reached at ktolly@ tolly.com.