Adventures in data recovery

Cybercrooks get eight-year sentence for extorting money from gambling Web sitesBut instead, they elected to pay the ransom.

On Friday, they called us back.

Apparently, they paid the ransom and when the Russian hackers attempted to decrypt the machines, four of them came across okay, but the fifth server – the one that had 60GB of Visa card information – blew up.

What was interesting is the Russian hackers spent about 12 hours working with the casino guys trying to solve the problem.

(I thought they would just take the money and run, but they didn’t.)

They kind of messed things up a little more.

On the Saturday, we were visited by one of the principals of the company himself, who hand-carried all the disk drives from that Visa card server and brought them to our lab.

He’s the only man who’s ever read every single word in our fine-print. This was very important to him. He didn’t sleep very much, but neither did we.

Seventy-two hours later, we learned that Exchange databases arranged the data with a serial number, so we wrote a program.

We did a little bespoke programming that stripped out all the records. Then we wrote another program that put them back together in the order we liked.

The problem was, when you encrypt something you need space – a temp file – big enough to encrypt. The Russians didn’t look and there wasn’t enough space on the original drive, so when they started to encrypt, everything blew up.

It took our two little bespoke programs to put all the bits back together again and the casino guy got on the plane 72 hours later.

Their loss was something in the neighborhood of $135,000 a day, so it was important he get back to work.

Venezuela mistook disk drives for organ transfers

A few years ago, there were some mud slides in Venezuela. The entire town was under – waist deep in mud.

We got a call from an IBM man who was down there attending to the crises. He alerted us he was sending three different servers – the disk drives only – to use for recovery.

What we were looking for was the engineering drawings, the sewer system, the civil engineering drawings – they had to find their city again.

But there was a mistake in judgment at the time of shipment. We couldn’t believe it – someone had elected to put all the individual disk drives into some baggies and, like organs being transported, they packed them in ice.

Five days later, it arrived in our lab, totally wet.

It took us quite a long time, going drive by drive to recover all the data bit by bit, but we did get all the drawings back – better late than never.

Toronto hospital needed ER treatment for RAID array

A few years ago, on New Year’s Eve, we got a call from Baycrest Hospital in Toronto. They had a 25-disk array running their pharmacy data – the whole hospital – that had run into some difficulties.

They struggled with the difficulties for about three days and couldn’t resolve the problems. They called in the manufacture of the RAID, who sent two engineers to the site in this emergency scenario.

When the engineers arrived, they did the best they could. They took all their years of experience and creative thought. They tackled the problem and reinitialized the server and invoked something they had no knowledge of – the scrub function.

After about 10 to 15 minutes, one of them said, “My God, what have we done?” He powered off the RAID array. It was a good thing he did.

Both of the men went upstairs and resigned from their jobs, thinking they had made things totally unrecoverable.

The entire 25-drive array showed up at our lab about eight hours later. They also sent with it their tape backup.

(We always ask, “Where is your backup? What’s wrong with it?” Sometimes the backup is old, sometimes there is problems with it, but some data is better than no data.)

We began to do our job recovering. It takes a long time to really scrub all 25 drives. Yes, they scrubbed quite a bit of it, but there was still data there not easily accessible.

As the team began to recover from the servers the best they could, we began Plan B, which was to work on the tapes.

They had actually taken a sample of the data that they tried to restore and sent it to the tape drive manufacturer. They got an instant reply and the instant reply had diagnosed the data as junk.

However, at midnight in our lab, with half the team working on the server, the other half started to look at the tape. Lo and behold, what we found was an old backup of the target data.

This old backup was very useful because by dawn, all we could recover from the 25-drive array was about 85 per cent of the target data. But you put the two together, and we had 100 per cent of the target data by 9:00 a.m. the next morning.