A research company in the financial services field recently discovered that it takes only one unethical person to single-handedly cause a major corporation a lot of stress and money by hacking into the company’s Web site. As the bad publicity was the most damaging part of the incident, the company’s chief technology officer (CTO) agreed to let IT Focus readers learn from this experience, on condition of anonymity. Here’s his story.
“The original point of penetration for us was Sept. 18, 2001, the day the Nimda virus went out and hit 2.2 million or so servers in North America. We were attacked by that. However, under advice by our network management company, we applied a patch. The patch solved the problem; we rebooted the servers and everything seemed to be fine. (See lesson 1.)
“In December 2001, we received communication from a hacker that said basically ‘I broke into your site, let me help you fix it.’ At that time, we were okay with the whole process, thinking ‘okay, there are ethical hackers out there, maybe it’s okay.’ In talking with this person, it became immediately apparent that it was in fact a job interview. He wasn’t offering to help us, he was willing to bill us for his services to close this hole.
“So, we ran a security check (of our Web site) with a company. We found somebody that was recommended to us and we were satisfied at the end of it – ‘yea, everything’s fine.’ (See lesson 2.)
“When we found out this guy was just a sales pitch and we’d done our security scan, we ignored him.
“In January, he came back to us and said ‘this is your final warning, I’m going to report you to the media if you don’t pay me to solve the problems.’ Since we had in our hands something from a security company saying ‘there are no holes’, we told him: ‘this isn’t going to work, we’ve called your bluff, take off.’ (See lesson 3.)
“He looked up all the press releases we’d sent out with all the customer names and he sent out an email to (the media and) 50 of our customers and said: ‘(Name of company) has a security hole; I was able to do this and I was able to do that.’
“This generated a flurry of customer calls coming in and saying ‘what’s going on?’ That hurt our reputation. This is the most damaging part.”
Of course the media was thirsty to report on security leaks and the news hit CNN and ZDNet.
“After the media blitz that he sent out, we received attacks from 36 other hackers, all trying to penetrate our servers. It was open season on our company for about two weeks. We were under denial of service attacks, buffer overflow attacks, Nimda attacks… everybody said ‘this is going to be good, let’s shut them down!’ During that time we were able to keep the site up 100 per cent. It was a lot of work.
“We even had an attack on an employee’s personal computer. That attack tried to go through the VPN into our network. They knew enough about this company that they knew this person worked at a home computer and was an authenticated user. They weren’t able to get in through the VPN but that’s where they could have caused real damage.” (See lesson 4.)
“Luckily for us, we were able to show that there was no information that was compromised, that we had run a security scan and that we were in the process of hiring a more reputable company. For us it died down as quickly as it started up and we had such a good relationship with customers and a good reputation prior to that, that we were able to recover. If we were a public company, I think the damage would have been much more extreme.”
And what about the hacker? This spring, a man arrested and confined in jail for a night by Toronto police was charged with five federal offences. At press time, his case had yet to come before the court.
With clear vision that only comes from experience, here are the firm’s learned lessons.
Lesson 1. “We should have gone back a week or even a few days later and applied the patches because those (later) patches were actually more complete than the previous patches. We saw Nimda patches and said ‘we’ve already installed that, we don’t need to put it in again.’ Initial patches for viruses often don’t close all the holes, just the symptoms.”
It was the firm’s policy to install Microsoft patches only after the patches had aged about a month since Microsoft’s brand new security patches had in the past caused problems with other people’s servers.
“At least for a time, we weren’t able to do that anymore because as Microsoft releases patches – and that’s more than one a week – anybody monitoring that service may have enough information to attack again. If you’re threatened, I think it is probably better you install the patches right away because you have a very real threat versus a relatively minor possibility of having problems with your servers. If you’re in a position where you haven’t had a threat, I would stand behind that aging period and give it a month before you install patches before you have any problems with your servers.”
Lesson 2. Be sure you can count on the security company that you deal with. “They missed a hole that was opened by the Nimda virus. The patch that we installed on Sept. 18 did not close it. Later on, we found out that this would have been one of the first things a security company should have checked.”
The holes were related to a directory on the company’s Web servers called scripts and some other directory names that still had execute rights. The hacker was writing command prompts from the script directory even though the cmd.exe file was not in that directory. “He was able to get into the Windows NT system 32 directory and run anything he wanted. He was actually running FTP (file transfer protocol) programs every night on script on his Web server – just calling our Web server, running FTP and downloading the Nimda virus every single day. We weren’t checking our logs. We’re a very popular site and probably get 20 million hits a month. To look through those logs for anything strange is quite difficult. We basically didn’t bother.”
To reassure users about the safety of its Web site, the company subsequently spent much more money on a more thorough security audit with “a new security company that has been in business for 15 years and their clients are our clients and I’ve checked their references in about six different places.”
Lesson 3. “If you’re being threatened and you feel the hacker may release it to the media and you have a public reputation, it may be better to buy him off than to risk the public problems. Of course that’s not a good long-term solution but it will help you out. His M.O. was to pose himself as a consultant. It’s only when we decided ‘thank you very much for the warning, however we’ll hire someone that we’re more comfortable with to do it’ that his true motives came out. Had we known that spending $10,000 to have this hacker come in and fix this one problem, may have saved us a few weeks of nightmares…”
Lesson 4. In addition to securing your network through firewalls, if you use VPN (virtual private network) clients, you also have to secure the home computers of employees’ that have access. “That’s a back door for them to come directly into your network as an authenticated user which is extraordinarily damaging. We now have a policy that if you have VPN client at home, you have to have Microsoft patches installed, licensed anti-virus, that type of thing.”
“There really isn’t a sure-fire way to protect yourself,” adds the CTO. “Because Microsoft continues to release patches, it doesn’t matter if you’re up to date today. By tomorrow you may not be. Often those patches are discovered after someone’s cracked into it, so during that period of discovery and certification and for them to write the patch, you’re vulnerable and you don’t even know it. I’ve since subscribed to several list services that send out these types of warnings so at least I can be aware of what the current situation is with our equipment.”
One of these is NTBugtraq run by Russ Cooper out of Lindsay, Ont. ( www.ntbugtraq.com) NTBugtraq is a mailing list for the discussion of security exploits and security bugs in Windows NT, Windows 2000, and Windows XP plus related applications.
In fact, editor Russ Cooper, a.k.a. “surgeon general” of Internet-related security solutions vendor TrueSecure Corp., was contacted by the hacker. Cooper then informed the firm of the hole the hacker had found.
Support also came from the police when the firm contacted them in January. “They took this very seriously,” says the CTO. “They were excellent from start to finish but they were tied in that it was his word against our word until he actually did something.”
It also doesn’t hurt to have contingency plans for threats and bad press. “It didn’t matter whether or not we’d done a security scan. It didn’t matter if there wasn’t even a hole to be found. All that didn’t matter,” marvels the CTO. Just the public accusation of a security gap was damaging.
