We test lots of network devices – Fast Ethernet/gigabit Ethernet switches, routers, VPN switches, firewalls – often measuring how many packets are dropped (zero-loss throughput) as a key gauge of overall performance. During the past few months, many of our customers have asked if we can alter the acceptable loss tolerance in our zero-loss test suites to something less stringent. My answer to this question has been a firm “no.”
The reasons behind these requests have varied.
One vendor said the product being tested was for “residential users” and that these users (read: you and me) didn’t understand or care about the difference between 0.001 per cent loss and one per cent loss.
I found it alarming that this vendor (and so many others) are looking to lower the testing standards instead of raising the quality of the products, especially in the small-office/home-office market. Just because I’m a residential user, does that mean I don’t deserve the same quality I have come to expect from my workgroup and enterprise products?
May I remind everyone that according to RFC 2544, the benchmarking methodology for network interconnect devices, zero loss means that no packets are lost. Currently, The Tolly Group’s de facto standard for zero-loss testing is five nines (99.999 per cent) of throughput or 0.001 per cent loss. Although The Tolly Group has on one or two occasions lowered the bar even further, it was done not because we wanted to, but out of necessity to obtain comparable data points in the products being evaluated. Even then, we kept the more stringent zero-loss results for the product that performed at the 0.001 per cent rate. Although rare, there may be times when lowering the zero-loss bar is necessary.
Let’s take a look at what the difference between 0.001 per cent and 0.1 per cent loss really means.
When we test a device with a loss tolerance of 0.001 per cent what we are saying is that out of every 1,000 frames that traverse the device under test, only one frame is allowed to be dropped. When we test at a 0.1 per cent packet loss, of the same 1,000 frames that traverse the device under test, we are now saying it is acceptable to drop 100 of these frames.
OK, so what does it mean to me? Imagine you are downloading a 6Mb file – we will use one that is 6,373,920 bytes. Using this file and the largest standard Ethernet frame size of 1,514 will require more than 1,400 frames to complete. What happens if we lose one frame? Probably nothing, but as the frame loss increases, so do retransmissions and timeouts that ultimately can lead to session termination. This packet loss is definitely unacceptable when performing secure transactions where state must be maintained on the sessions.
What can we do? You can insist vendors offer testing results that are fair and of the highest standards – meaning, don’t play games with packet loss tolerances. Make the vendors raise the quality of their products instead of lowering the standards to help make their products look good.
Tolly is an engineer with The Tolly Group in Manasquan, N.J. He may be reached atbtolly@tolly.com.