The addition of a new software routine caused the multi-hour failure of Research in Motion Ltd.’s Blackberry messaging service in North America earlier this week, the company said Friday.
In its first statement explaining the outage, which began on Tuesday evening and continued until Wednesday morning, the company said the introduction of a new software routine that was supposed to optimize system cache memory led to the problems.
“The system routine was expected to be non-impacting with respect to the real-time operation of the BlackBerry infrastructure, but the pre-testing of the system routine proved to be insufficient,” Research in Motion (RIM) said in a statement.
Things were made worse by the poor performance of the company’s back-up system.
“Although the backup system and failover process had been repeatedly and successfully tested previously, the failover process did not fully perform to RIM’s expectations in this situation and therefore caused further delay in restoring service and processing the resulting message queue,” the RIM statement said.
In pointing to the software fault the company also said it had ruled out any security or capacity issues or the failure of hardware or core-software.
RIM, which apologized to customers for the outage, said its examination of the problem was not yet complete and that more information would be provided as it becomes available.
“RIM has already identified certain aspects of its testing, monitoring and recovery processes that will be enhanced as a result of the incident and in order to prevent recurrence,” it said.
During the period of the outage, callers to the BlackBerry’s technical support line were still greeted with the following message early Wednesday morning: “We are currently experiencing a service interruption that is causing delays in sending or receiving messages. We apologize for the inconvenience and will provide updates as soon as they become available.”
New York television news channel NewsChannel4 reported Tuesday night that the problem affected “all users in the Western hemisphere.”
However, comments from operators in Asia and Europe, as well as postings to the BlackBerry Forums, suggested the problem was limited to North America.
RIM officials told NewsChannel4 that they were trying to reset the system and expressed concern “that the backlog of data, which will rush through when it comes back on line, could cause a bigger problem,” the news channel reported on its Web site.
RIM officials advised people who use BlackBerry as a major way of communications to make back-up plans, the channel reported.
Initially there was speculation that the outage may have been caused by one of RIM’s Network Operating Centers (NOC) going down – something that has reportedly happened before.
RIM operates two NOCs, both located in Canada, and the company has considered locating additional NOCs outside of Canada, noted Emma Mohr-McClune, principal analyst with Current Analysis Inc.
Companies that provide BlackBerry service connect their mail servers to a BlackBerry Enterprise Solution (BES) server located on their premises, which in turn is linked to one of RIM’s NOCs, according to Mohr-McClune. “All data slides to Canada and back,” Mohr-McClune said.
Other parts of the world appeared to have been unaffected by the outage. A representative for Taiwan Mobile Ltd., RIM’s BlackBerry partner for the island, said the problem is limited to North America, and that users would not be affected unless they are sending or receiving e-mail through a BlackBerry server there.
“RIM has not communicated with Taiwan Mobile about when this problem might be fixed,” said the representative, April Hong.
NTT DoCoMo Inc. in Tokyo said its BlackBerry users in Japan were also unaffected. And In Europe, a spokesman for T-Mobile Deutschland GmbH was unaware of any problems, and Blackberry users in Germany and France reported no interruption of service.
The episode occured at a time of continued rapid growth for Waterloo, Ont.-based RIM.
The company added 1.02 million subscribers in the quarter ended March 3, for a total of approximately 8 million BlackBerry subscribers worldwide. Revenue for the quarter was $930.4 million, up 66 percent from a year earlier. Net income for the quarter before adjustments was $187.9 million, the company said.
