The one-day shutdown of the TSX and TSX Venture Exchange offers just as many important lessons for IT organizations as it does for Canadian traders, according to industry observers.
Shortly after opening Wednesday morning, tech workers at the Toronto-based stock exchange noticed some data was missing from the exchange’s computer systems. While the system could still process orders, traders did not have access to real-time data such as stock quotes, corporate actions, and other index information.
The stock exchange ultimately decided to shut down trading just 18 minutes after the opening bell and did not resolve the issue until Wednesday afternoon.
TMX Group Inc., the company that runs the TSX and updated traders on the shutdown several times through the day, said its initial findings indicate “a network firmware issue” resulting in “complications with data sequencing.” In a statement, TMX added that the glitch impacted the delivery of Level 1 data feeds. The back-up systems were also affected, the company said.
“The TSX is a utility,” Rob Burbach, senior analyst with IDC’s Financial Insights Canada, said. “But the difference between a utility like electricity and the stock exchange is that we don’t have any alternatives for our electricity. We certainly do have alternatives to trade stocks.”
And many traders did just that, flocking to alternate trading systems, including the New York Stock Exchange to make trades on inter-listed company stocks.
Michael Ptasznik, chief financial officer with the TMX, told news outlets that the system runs on “very complex technology” and despite its best efforts, the TSX will not always be able to maintain 100 per cent availability.
But according to some analysts, for a service like the TSX which needs to be reliable and reputable in the eyes of its users, that kind of availability is expected – especially during trading hours.
“To be fair to the TSX, it’s probably something you can get away with once, but the fact that it took all day to fix was where it became more of a problem,” Burbach said.
He added that earlier this year, a software problem caused a similar shutdown at the London Stock Exchange.
“But even in that scenario, they managed to get it up before the end of the day and people were able to get their trades in before the close,” he said. “We didn’t see that opportunity in Toronto.”
Carmi Levy, a technology analyst with AR Communications Inc., said that the real impact of the TSX meltdown will be felt over the next few months as publicly traded companies and their stakeholders continue to evaluate where it makes the most sense to be listed.
“The TSX has just given them a perfect excuse to begin considering other exchanges, and nothing short of an all-out investment by IT to re-establish the exchange’s technical credibility will stem the tide,” he said.
Although Levy agreed that the TSX can’t promise 100 per cent availability of every online system, he felt the exchange could do a better job of building redundancies into its systems to prevent a complete meltdown.
“It’s one thing to have a failure, followed by failover to a redundant system. It’s quite another to have a failure followed by complete nothingness,” he added.
As for what businesses can actually learn from this, Levy said that companies will want to ensure they aren’t similarly exposed to a total system meltdown without an alternative or workaround. That means dusting off your disaster recovery plan and ensuring you anticipate a wide range of potential outage scenarios, specifying how the organization can respond and the resources required to carry out the response.
“[In the TSX’s case] the back-up plan to a scenario like this must incorporate some sort of redundant trading system availability in the event their primary systems are unavailable,” he said.
“If they had a fallback plan, it wasn’t in evidence in this case. So while it’s unrealistic for any one system to come with a 100 per cent availability guarantee, it’s equally unacceptable for something like this to cascade into the kind of widespread, long-term event that we saw this week.”
But while the 24/365 goal is a grand focus, it is also an unrealistic and unnecessary one for many IT shops, according to a recent Info-Tech Research Group report on data centre reliability. In the case of an e-commerce site, the bulk of activity might occur during the morning or early afternoon. If only two per cent of Web traffic occurs outside of those times, it gives IT a clearly defined window for maintenance and upgrades.
The research firm added that most enterprises are working towards the “high availability” model rather than a “continuous availability” model.
