People working in the Assay Office during the California Gold Rush were obliged by law to wash and dry the scale, avoid breezes by making sure the windows were closed and verify the scale’s accuracy with an object of known weight before each use. Today, McDonald’s Corp.’s restaurants have similarly stringent rules for weighing precooked Quarter Pounders.
In our labs, we’re just as careful about weighing things. Instead of gold dust and hamburgers, we balance network hardware and software on the scales. We have standards for creating a level playing field when testing products head to head.
When we test with Windows NT or 2000, we make sure background processes, disk utilization and file position, network configurations and even monitor resolutions are the same across all our tests.
Services running in the background are a big factor in configuring a fair contest environment. Take Microsoft’s Index Server process (cisvc.exe), which prepares Web-accessible indexes for use by a search engine. Because its periodic background examination of files can be momentarily disk- and CPU-intensive, Index Server can profoundly influence the performance of a product we’re trying to test.
Other Windows services offenders are the Distributed Transaction Coordinator, World Wide Web Publishing Service, File Replication Service, Network News Transfer Protocol, SMTP, Network Dynamic Data Exchange, DNS, Dynamic Host Configuration Protocol and the Windows Media services. We make sure all Windows services – if the product we’re reviewing doesn’t need them – are turned off.
A protocol analyser trace confirms each test occurs in as quiet an environment as possible.
When we’re testing multiple products on the same computer, we reinstall and reconfigure Windows for each product. Just uninstalling one product before installing the next isn’t enough. Most uninstall procedures leave at least some residue in the computer in the form of obsolete hidden files, registry entries or system configuration changes. The residue might cause the next product’s files to install in different locations on the disk, which can sometimes affect performance. Moreover, if a product sets up a new Windows service or replaces some of Windows’ system files, the uninstall procedure often leaves these components on the computer.
If we run tests on multiple computers, we use duplicate machines for data storage, business logic and presentation logic or software agents.
If one product requires a monitor resolution of at least 800 by 600 and another requires at least 1,024 by 768, we configure Windows to use the higher resolution for all tests. Video drivers can consume considerable CPU time, especially at higher resolutions.
Creating a fair testing environment on Unix or Linux presents basically the same challenges as on Windows, but the terminology is different. We remove entries from .RC files that start unnecessary background processes (such as daemons) and reboot.
Following these guidelines can help ensure your testing is always fair and accurate.
Barry Nance, a software developer and consultant for 29 years, is the author of Introduction to Networking, 4th Edition and Client/Server LAN Programming. He can be reached atbarryn@erols.com.