Info-Tech Research Group Ltd. has unveiled its top five automated performance testing tools with HP Inc. and IBM. Corp. leading the way.
For large enterprises, HP’s newly released LoadRunner 9.5 can be considered a market leader, according to Howard Kiewe, senior research analyst with the London, Ont.-based consultancy. The product scored high marks for its solid integration with HP’s Web-based load testing and monitoring application Performance Center, as well as its strong support for Web 2.0, rich Internet applications, Flex, and AJAX.
“HP is pretty much best of breed in this space, but it’s the most costly solution,” he said, noting that HP does not publicize its pricing details and has a reputation for being very expensive. Kiewe added that LoadRunner is Windows-dependent for test scripting, which is not advantageous for IT shops that work with other operating environments.
But while enterprises with less complex QA requirements might want to avoid the hefty price tag, the HP offering is the most comprehensive app testing tool on the market, according to Info-Tech.
IBM’s enterprise level tool also ranks as a market leader, Kiewe said, especially as the need for automated Web and network-driven application performance testing tools grows. The company’s IBM Rational Performance Tester 8 is based on the IBM Jazz platform and supports fully Web-based testing.
“That’s going to be good for a shop doing Java-based development, because of its close ties to the Eclipse IDE (integrated development environment),” Kiewe said, adding that IBM’s offering comes at a lower price point than the HP solution.
Other strengths include support for a variety of Web protocols and application types such as HTTP, SAP, Oracle and Citrix, as well as compatibility with Windows, Linux and z/OS. On the downside, Kiewe said, the tool is heavily focused on load testing, meaning QA shops might have to purchase additional tools to address other test types.
Rounding out Info-Tech’s top five are Borland Software Corp.’s SilkPerformer, Micro Focus Ltd.’s QALoad, and Microsoft Corp.’s Visual Studio Team System 2008 Test Edition and Test Load Agent.
Despite its compatibility with platforms such as .NET and Java, integration with other vendor tools and useful tiered pricing option, the Borland offering is hindered by a cloud of uncertainty, Kiewe said.
“They are in the middle of acquisition talks with Micro Focus and another undisclosed company,” he said. “They’ve also been bleeding money for years. Not because they’ve had bad products, but they’ve just had some difficulty executing at a business level.”
Micro Focus, which acquired Compuware Corp.’s automated software testing business this year, received top marks from Info-Tech for its compatibility with multiple operating environments and a variety of Web and network-driven applications.
But, according to Info-Tech, it’s involvement in the Borland deal probably means that IT shops should hold off on choosing either solution for a few months until the acquisition details become clear.
Microsoft’s offering rounds out the top five, with high marks being given out for its strong integration with other Microsoft-based projects and its per-CPU virtual user licensing model.
“Microsoft is the most different offering in the batch in that it’s really only worth considering if you’re a Microsoft shop and you’re already using Visual Studio,” Kiewe said. “The level of integration that you get with the Visual Studio software development kit is one of the things that really make it standout for those people.”
For organizations doing Web or Java development, the tool will probably not be as effective as the other offerings in the top five, he added.
But despite the vendor scorecard, Info-Tech advised organizations to establish their internal quality assurance processes before going on a shopping spree. Kiewe said that organizations often underestimate the complexity required to setup a performance testing program.
“There’s a kind of overly optimistic desire that you just buy the tool and it’s going to solve your performance problems,” he said. “A tool is a way to facilitate a different type of work flow in development that will eventually lead to a faster and more stable piece of software.”
Kiewe added that companies must first develop a clear QA strategy that focuses on defining your test processes and having the right people in place to lead those initiatives. This includes deciding on the right Software Development LifeCycle model and the primary application languages and platforms you want to develop on.
