STRATFORD, Ont. — There are two main lessons for CISOs from last April’s ransomware attack on the Ontario cottage country town of Wasaga Beach, which was forced to pay $34,950 in bitcoin to get access back to four of its servers:
First, have a backup server that’s separated from the network. If it isn’t, it will be infected. And with ransomware there’s a good chance it can’t be decrypted.
Second, have your disaster recovery plan on paper, not stored only online where lesson number one will come into play. (Side lesson: Don’t throw out those whiteboards)
And arguably there’s a third lesson: Don’t take a day off. Because that’s when Derek Bowers, the town’s chief information technology officer, got a text message from his technician: just after 8 a.m. The finance department couldn’t get into the payroll system and some files.
Bowers recounted the incident Tuesdsay during an interview at the annual security conference of the Municipal Information Systems Association of Ontario.
He’d never seen anything like it in his 15 year IT career.
Dashing back to the server, Bowers issued instructions to staff to disconnect the town from Simco County’s network, lock down ports on switches, turn off systems showing they were not infected and to detach the town’s 124 workstations from the Internet. Six of them, which had been left on during weekend, were infected with what they later learned was version two of the Dharma Crysis ransomware. Unlike version one, there was no decryption solution available.
Initially it looked like only one server had been infected.
But as Bowers started going through each of the physical and virtual servers. one at a time. “As I went through each one it just got worse,” he recalled, realizing six physical and 11 virtual servers were encrypted, too –including the backup, mirror, file and SQL servers. “It was a gut wrenching feeling … but it was one of those situations [where] you can’t panic.”
Other than its Exchange 365 cloud email service, all data and software were on-premise.
After letting management know of the situation, the town’s CAO asked Bowers to reach out to others for help. These included Compu-Solve Technologies in Barrie, Ont., which helped rebuild the network, consulting firm ISA Inc., which tried to decrypt the locked servers, a forensic analysis firm called Hexigent Consulting of Oakville, Ont, and the new cyber forensic unit of the Ontario Provincial Police. Simcoe County also sent help.
After more than 12 hours, Bowers went home to try and get some sleep; he was back at it at 4 a.m.
On the second day the workstations not impacted were re-imaged and the network rebuilt.
Ironically a network overhaul this year had been planned for, including more robust subnets, but, Bowers said, “we were behind the curve” in getting it down. As a result, “we didn’t have the air gap that we needed on our backup server.”
Slowly service was restored. Departments had at one laptop with internet service through a mobile phone or mobile hotspot so they at least had email, which wasn’t connected, a productivity suite and a printer. Each re-imaged 118 workstations was thoroughly scanned before being put online. After two weeks the majority of them were back online, but they couldn’t access any data as new file structure had to be rebuilt.
However, the municipal payroll is still being done on spreadsheets, largely because the new computerized version isn’t ready yet.
Outside of the town hall, the firehall, public works department and the recreation centre had limited services because they were on a recently installed separate Internet network.
Meanwhile, a decision had to be made about paying the ransom. Most of the servers could be rebuilt, but Bowers realized the data on four of them were essential. Those were the ones the town decided to pay the ransom for to get decryption keys. And finding a way to buy bitcoin was a challenge.
About that disaster recovery plan: As mentioned earlier, it was locked on one of the system. Fortunately, Bowers wrote – and remembered — it. “My office had a huge whiteboard that had every server, what was on it, their internet addresses, what users had to be brought online first.”
The town figures the attack has cost it $251,000, so far. The incident is still being investigated, which, Bowers said, and the infrastructure is still being ironed out. The total cost could rise.
It was later determined that one of six workstations hadn’t been turned off for the weekend. One of them could have been the source of the initial infection from a phishing email, Bowers said. . However a device could have been infected weeks before.
In addition to making sure the backup system is fully isolated from the production network, Bowers advises CISOs, make sure your disaster recovery plan is well tested and includes all corporate staff. Users have to figure out how they will continue to do their work if IT resources aren’t available.
“We’ve changed up a lot of our structure so we can be back up in around 40 minutes,” he said.
“If you can mitigate the impact, that’s key.”
But it was an expensive lesson.
