A new measurement system recently announced by market research firm IDC could give supercomputer customers a new way to measure the performance of those systems and more accurately evaluate them before purchase, according to the company.
The IDC Balanced Rating is a new supercomputer and high performance computing (HPC) test that measures the performance of systems by testing their processor, memory and scalability performance, said Earl Joseph II, director of research for high-performance computing at IDC. IDC is a division of International Data Group Inc., the parent company of IDG News Service.
The tests are performed only on systems that are installed and running at customer locations, rather than at vendor locations, to ensure accuracy and that the tests will return real-world performance results, Joseph said. HPC systems are used in vehicle design, drug and genome research, weather forecasting and a number of other areas, he said.
That there was a need to document the real-world performance of these systems became clear to IDC after it formed the High Performance Computing User Forum, a group that includes 1300 HPC users, to try to better understand what issues users were facing, Joseph said. The members of the forum said that they needed a way to know what the actual performance of these systems was, not just the peak performance that other tests measure, he said.
“They wanted a way of looking at computers to understand better the way they were going to perform,” Joseph said.
The Balanced Rating, IDC’s answer to those users’ questions, tests the processor performance, memory capacity and scalability of 893 HPCs installed worldwide. The processor uses the Specfp_rate_base2000 benchmark to test floating point operations, the SPECint_rate_base2000 to test integer calculations and the Linpack Rmax test which solves dense linear equations, Joseph said.
The memory system capacity test examines how much data can be moved in and out of memory in a given period using the STREAM TRIAD test. Scaling capability is determined by using total processor count and total system memory and interconnect bandwidth.
The systems tested in this way have been broken down into four categories: Technical Capability Computers, which includes systems designed for the most demanding tasks; Technical Enterprise Computers, systems that cost over US$1 million; Technical Divisional Computers, systems that cost between $250,000 and $999,000; and Technical Departmental Computers, which are machines priced under $250,000.
The top five overall systems, as measured by the Balanced Rating, are the NEC Corp. Earth Simulator and IBM Corp.’s ACSI White SP Power 3, followed by Hewlett-Packard Co.’s LANL AlphaServerSC45 P-2, PSC AlphaServer SC45 and CEA DAM AlphaServer SC45.
The rankings are available free online at the HPC User Forum Web site, located at http://www.idc.com/hpc. The site also offers a custom filter of the information to allow users to rank systems based on their needs, Joseph said.
The Balanced Rating currently includes six tests, but Joseph would like to increase that number to 12 in a few years. IDC will be working with the U.S. Department of Defense and the San Diego Supercomputer Center to build new tests, he said.
The performance numbers will be updated every quarter, and, starting in November, the list will also include the price of each of the systems tested, he said.
“Our goal is to get to a better way of calculating price/performance,” he said.