DALLAS – Glen Wize, a senior systems administrator with Toronto-based Teranet Inc., is sure that his firm’s disaster recovery (DR) plan is up to par. Teranet, which stores digital property records for the province of Ontario, deals with critical info and must take data back-up seriously.
“It’s mandatory for our work,” Wize said. “Property information is very important to the province of Ontario. It has to be treated like a bank.”
With such crucial data in its care, Teranet devised a multi-faceted strategy for the worst-case scenario in IT, Wize said. Even if all of the hardware imploded, the company would be able to carry on.
“Every product we put in, we have to have an approved disaster recovery plan before it hits the floor,” Wize said.
Teranet also performs a data storage integrity test daily, to make sure disks, servers, software and tape media are working right.
Considering the company’s extensive safety procedures, “we have a high confidence level,” Wize said. But judging from comments garnered at a data storage conference in Dallas in April, where Network World Canada interviewed Wize, his certainty might be misplaced.
Veritas Software Corp., a Mountain View, Calif.-based storage software maker, produced the gathering, dubbed Vision 2002, where the company’s customers learned a thing or two about DR planning.
Greg Valdez, Veritas CIO, said just 12.5 per cent of enterprises would successfully recover from disaster. Although many enterprises have DR plans (28 per cent), they are not prepared for large-scale worst-case scenarios.
Valdez wondered how many of those DR blueprints are tried and true. How often do enterprises test their plans? Are the procedures foolproof and well-documented?
Not likely, Valdez said. After all, when it comes to spending time and money on DR, most companies prefer to put effort into revenue-generating ventures. “Would I rather have my people focused on a disaster that may or may not happen, or do I want them out there making sales?”
But if a company’s data storage technology is offline, the sales reps could have a tough time referencing customer information and ultimately closing deals. Gary Bloom, Veritas’s president and CEO, said 40 per cent of companies that have no DR plan disappear within five years of inception.
“Information has become king,” he told the conference audience. “And with it, storage has become mission critical.”
That said, Brooks Graham, Veritas’s senior product specialist, offered ways for storage managers to move from “panic” to “planned” should the worst befall the data centre.
Graham said it’s important to understand the difference between myriad recovery methods. Back-up tapes offer restoration only up to the most recent recording. If you vault once a week, you could lose a week’s worth of data. Periodic data replication can save information that was replicated mere hours before a disaster. Synchronous replication provides almost immediate recovery.
Apply the right technology to the right data, he said. Use replication for e-commerce information so your money-making Web site can be up and running as quickly as possible. Use less speedy back-ups for printer spool files.
Keep in mind that back-ups and replication are not the same thing, Graham said. “Backup is data protection.…Replication is not.”
Let’s say a server goes down because of corrupted data. Chances are, the most recent replication copied that corruption. With a backup, you could rely on an older, uncorrupted version of data to restore the box. But without that legacy snapshot, restoration may be out of the question.
With tape back-ups, vault often and elsewhere. Do not keep your tapes on site. If the site is destroyed, so too are the tapes.
Operate two data centres, a main site and a failover site, Graham said. Make sure the locations are some distance apart. That way, a natural disaster is less likely to affect both sites.
Documentation is an integral part of DR planning, Graham said. Write down step-by-step instructions to tell operators what to do in case of a disaster. “Implementers shouldn’t think, (they should) just do,” he said. After all, thinking isn’t easy during times of crisis.
Stephen Elliot, research director with Framingham, Mass.-based Hurwitz Group Inc., said he is skeptical of most enterprises’ disaster readiness. Testing and documentation in particular are “huge question marks. Talk to 10 companies;…the majority probably don’t have complete plans.”
But when asked, conference goers said they employ DR planning.
“I’m sure everybody has plans in place,” said Bruce Froelich, a member of the LAN operations centre at Aetna Inc., a healthcare benefits company in Hartford, Conn. “But how feasible those plans are during a disaster is another situation.”
For example, Aetna attempted to restore a pair of servers during a recent DR test. “We passed, but it took much longer than expected,” Froelich said. The server software, designed for hardware so old that it’s no longer available, didn’t work on the new boxes. The Aetna team had to make unplanned and time-consuming changes before the servers operated normally again.
Teranet’s Wize admitted, “We haven’t had to rebuild all of our servers at once.” But asked if the team would be able to do so if required, Wize said it would.
Despite concern that many companies are not ready for disasters, Wize maintained his confidence in Teranet. The company has already applied certain best practices in DR, he pointed out.
Teranet has specific documentation for getting hardware running, applications back online and communications restored as quickly as possible. Also, the firm operates two data centres: one in Toronto plus a failover site in Mississauga, the next city to the west.
Teranet’s exhaustive DR planning puts the firm ahead of the game, Wize said. “It’s been reassuring,” he said. “The way we duplicate and off-site our data, these are the strategies that they’re promoting.”
