You can’t survive without them. They wield enormous power over your systems, networks, and data — the very lifeblood of your organization. Few people outside IT have any understanding of what they do, and fewer still exercise any oversight over their actions.
To be sure, the overwhelming majority of IT admins are honest, hard working, and underappreciated. But when they go rogue, bad things happen. Organizations find themselves locked out of their own networks. Customer data files inexplicably vanish. Companies scan their networks and discover somebody’s running a porn site from inside their data center. Trade secrets get destroyed or stolen, and employees get the creepy feeling somebody is watching everything they do — and they’re right.
Those are just the cases you hear about. Most companies do everything they can to keep news of rogue admins quiet, because the damage to their reputations could be even greater than the havoc wreaked by disgruntled or overzealous geeks.
Off the Record submissions
And many companies are virtually helpless to do anything about it, says Steve Santorelli, director of global outreach for security researchers Team Cymru.
“It doesn’t matter if your systems are utterly bomb-proof and you’re patched up the wazoo with nuclear-grade security,” he says. “A rogue system administrator with root or privileged access can bypass all your perimeter security and your tripwires, because they have to get into the system to do their jobs. The persons responsible for carrying out insider attacks are often the same ones responsible for spotting and preventing them. They know how to overwrite the firewall logs or change their access controls so that no one else can get in. They know where the backup logs are kept and how to manipulate their encryption keys.”
You may already have rogue admins in your organization, ready to blow. Here’s how to spot them and what you can do to minimize the damage.
Rogue IT admin No. 1: The crusader
He knows what you should be doing and how you should be doing it — and he’s not afraid to take matters into his own hands if you don’t agree. A well-intentioned but overzealous admin can often do as much harm as a malicious one.
There are lots of rogue activities that don’t involve disgruntled employees, says Josh Stephens, head geek for SolarWinds, maker of network management software.
“A rogue admin could simply be someone who chooses to do things his way instead of the company’s way,” he says. “Say your organization has standardized on Windows, but your rogue guy loves Linux. Three months down the road, you may discover that a third of your servers are now using Linux.”
Sometimes, though, when the crusader takes over, destruction results. Back in the mid-’90s, Jon Heirmerl worked for a software developer on a government contract.
“We had one network administrator — I’ll call him Jim — who would walk the halls looking for people who left their desks with their terminals still logged on,” says Heirmerl, who’s now director of strategic security for Solutionary, a managed security solution provider. “If Jim found a terminal still logged on, he would go into that person’s system and delete all their files to ‘teach them a lesson.'”
Then one day a senior developer caught Jim in the act as he was deleting files. The developer, who had no recent backups and lost months’ worth of work an instant after Jim hit the Delete key, went postal.
“He punched Jim in the face,” says Heirmerl. “Jim didn’t delete any more files after that.”
Perhaps the best-known crusader is Terry Childs, the former network administrator for the City of San Francisco who refused to surrender passwords to key city systems because he felt his supervisors were incompetent. Childs was convicted of violating California’s computer crime laws in April 2010 and is now serving a four-year term in state prison.
“It’s fair to say [people like Terry Childs] think they’re doing the right thing,” says Santorelli. “Hitler also thought he was doing the right thing. Just because you feel justified isn’t a defense for criminal acts. Most people would argue there are sufficient safeguards that allow you to be a whistleblower without restorting to destruction, whether it’s the media, government, or some regulatory agency.”
Anti-rogue defense: You can limit the damage individuals can do by implementing separation of duties and two-person controls, says Ken Ammon, chief strategy officer at Xceedium, a maker of appliances that manage how privileged users access key systems. That will ensure that sensitive tasks are performed by multiple people, and the same individuals don’t have responsibility for both performing tasks and auditing how they’re performed.
Rogue IT admin No. 2: The entrepreneur
You’d think keeping the lights on, the servers running, end-users happy (or at least not mutinous) and protecting the network from hackers and hooligans would be more than a full-time job for most admins. And yet, there’s the occasional rogue who decides to open up a little side business at work — on company time and using company equipment.
Heirmerl says he’s encountered rogues using company servers to sell everything from pirated satellite equipment to tarot products. In the latter case, the entrepreneur’s retail operation was discovered after he’d been laid off, and his replacement had unraveled the complex firewall rules the rogue created to allow him access to the network.
“Within 30 minutes after the firewall rules had been changed, the first admin called to complain that his access had been cut off,” he says. “This was two weeks after he’d been let go. He was very insulted and thought it was totally unfair.”
Winn Schwartau, chairman of smartphone security company Mobile Active Defense, says he was doing independent consulting for a financial services company in 2003 when it discovered one of its sys admins was running a fee-based porn site on his work desktop, using an external modem and a partitioned hard drive. The modem was discovered during a routine scan of the network for rogue communications devices, which led them to the porn site, Schwartau says.
The problem in cases like these is that no one else is watching, says Heirmerl.
“These people are not responsible to anyone,” he says. “The guy running the tarot site configured the system audit logs to hide his behavior. They’ve got all the authority and no accountability.”
Anti-rogue defense: Access and network management tools can go a long way toward preventing rogue activities, says SolarWinds’ Stephens.
“There’s no reason not to build in a management system that will notify you when someone is accessing systems they shouldn’t or changing passwords, so you can investigate what’s going on,” he says. “Solid management software can protect you from these kinds of activities.”
Rogue IT admin No. 3: The voyeur
They have the keys to the kingdom, and sometimes they use them when they think no one’s around. Given their almost unfettered access to company networks, some rogue admins can’t help but snoop.
Josh Stephens says he’s worked with numerous sys admins over the years who’ve been fired for reading other people’s email — or worse. One day about five years ago, Stephens says he was running a WebEx demo for 30 executives, showing off how SolarWinds’ Netflow tool could let you see what any user on the network was doing at any time. During the demo he picked an employee at random — a tech admin — and drilled down on his desktop.
“We saw he was on Monster.com updating his résumé, he had a World of Warcraft session open, and he was running a terminal server session to access the computers at the company he used to work for,” says Stephens. “I tried to back out of there as quickly as I could, but everybody saw it. I felt bad for the guy but … he wasn’t working there much longer after that.”
Joe Silverman, CEO of New York Computer Help, says in 2009 his computer repair service came to the rescue of a public relations firm that was being stalked by a former IT admin. The employee would remotely access the company network when he thought no one was in the office and snoop around the desktops of employees, who were mostly attractive women in their 20s. He pawed through their photos, spied on their calendars, and bcc’d himself on all their emails.
“He knew their schedules, so he would access their computers while they were at lunch,” says Silverman. “If one of the women came back early they’d see the mouse cursor moving on its own, or they’d end up getting in a tug of war with him over control of their systems.”
Silverman says they managed to lock the IT voyeur out by changing the admin passwords and cutting off all his access privileges — and that’s where the matter ended. The owner of the PR firm didn’t want to pursue charges.
Sometimes when geeks go wild, they do more than just look. About eight years ago managed security services firm NetSec was called in to help a well-known magazine publisher identify a rogue admin, says Ammon, who was CEO of NetSec when it was acquired by Verizon in 2006.
That publication, famous for photos of attractive young women in bikinis, was running an online contest where readers could vote for the next cover model. But the admin hired to manage the back end of that contest had a different agenda.
He accessed the database containing the names and addresses of each swimsuit model and offered to rig the contest in each woman’s favor in exchange for sex, says Ammon, who is now chief strategy officer at Xceedium, a maker of appliances that manage how privileged users access key systems. The scheme was only detected after one of the models called the magazine and complained. Ammon doesn’t know how many models accepted the offer without complaining.
“The big challenge with insiders like this is they tend to be both highly intelligent and very familiar with your infrastructure,” he says. “They’re able to violate policy simply by the nature of their position, and they’re mostly unmonitored. The question then becomes who’s watching the watchers.”
Anti-rogue defense: Don’t just rely on background checks to vet potential employees, says Schwartau. Smart employers also run psychological profiles to understand each person’s motivations, proclivities, and weaknesses.
“Are they a good guy or a bad one?” he asks. “Are they easily swayed by sex or money? Where are their buttons? Every law enforcement agency does it, but corporate security is behind in this.”
Rogue IT admin No. 4: The spy
IT admins don’t merely control systems, networks, and databases; they often have access to trade secrets, intellectual property, and corporate dirt. A rogue may decide to use this information for personal gain, to benefit a competitor, or simply to blow the whistle on employers — and there’s often little a company can do to stop it.
Proving corporate espionage is difficult. Borland found that out in the 1990s after former vice president Eugene Wang jumped ship to Symantec, allegedly taking scores of proprietary documents with him. Wang and Symantec CEO Gordon Eubanks were indicted for theft of trade secrets, but the charges were later dropped. Borland sued both Wang and Symantec; the case dragged on for five years before both parties agreed to dismiss it.
Not much has changed. Heirmerl says he consulted with a manufacturing company in 2005 that laid off an engineer in its R&D department because he was impossible to work with. The same day the engineer was walked out of the building, he went to work for the firm’s chief competitor. Three months later, that competitor released a product that was virtually identical to the one his former employer was set to announce a few weeks later.
“By being second to market, that firm estimated it lost something on the order of $80 million in sales,” he says. “They sued the engineer, but they were unable to prove that he’d stolen that information.”
More often than not, though, rogue spies tend to steal information that will help them start up their own ventures, says Ammon. Such was the case of Sergei Aleynikov, the former Goldman Sachs computer programmer who was convicted of stealing proprietary trading algorithms from his employer. Aleynikov was sentenced to 10 years in prison last December.
Ammon says rogues may also become whistleblowers — like Private First Class Bradley Manning, the intelligence analyst infamous for leaking more than 200,000 state department cables to WikiLeaks in 2009.
“Whether misguided or not, whistleblowers like this are going to become a bigger risk over time, especially as the next, far more open generation takes over IT,” he says.
Anti-rogue defense: Restrict access to proprietary company information on a need-to-know basis, and make employees who have have access to sensitive data sign a confidentiality agreement that binds them even after they’ve left the company, says Heirmerl. This won’t prevent admins from going rogue with your information, but it may make them think twice.
Rogue IT admin No. 5: The avenger
Hell hath no fury like an IT admin terminated for what he or she feels is unjust cause. It’s the most common rogue admin story — and the scariest.
Years ago when he worked for an ad agency, Troy Davis hired a young sys admin because he was reputed to have “mad Linux skillz.” But Davis had to let the admin go after six weeks because he had accomplished nothing.
“A few days later a client called me to tell me his website was down,” says Davis, who’s now CTO at CoupSmart, a company that lets small to medium-sized businesses create coupon campaigns and distribute them via Facebook. “I logged into their server, and sure enough, every file related to the website had been deleted entirely.”
A search of server logs turned up the few history files the attacker had neglected to delete, which recorded his IP address, log-in times, and complete shell history. When Troy contacted the service provider that owned the IP addresses, it confirmed the recently discharged admin was the guilty party.
“The local sheriff paid him a visit and let him know how close he was to serving time in prison, had I decided to press charges,” says Davis. “We ultimately lost the affected client over the site deletion incident because they simply didn’t trust us any more.”
Schwartau says he was consulting with a financial services firm about six years ago that fired one of its database administrators after it discovered the DBA was using company computers to hack into systems at his previous employer. The problem? The DBA was the only person who knew the firm’s administrative passwords, and he refused to turn them over until his bosses promised to write him a good recommendation. The firm agreed.
“They could have involved the police, but they didn’t want the publicity,” he says. “They wanted it kept quiet so as not to encourage others to do the same.”
Stephens says he once worked for a major U.S. telco that fired a network engineer for violating its HR policies.
“The engineer got wind of what was about to happen, so before he was escorted out he changed all the passwords to our core routers and wouldn’t give them up,” he says. “It was ugly. It took us quite a while to reset everything and make sure he was actually locked out.”
But the champion avenger may be Roger Duronio, a former sys admin for UBS Paine Webber. Unhappy with the bonus he received, Duronio planted a logic bomb on 1,000 of the brokerage house’s computers in March 2002, designed to take them offline. He then shorted the company’s stock in the hope that negative publicity following the Paine Webber outage would drive its share prices down, putting money in his pocket.
It didn’t work, says Keith J. Jones, a senior partner with computer forensics firm Jones Dykstra & Associates, and an expert witness in the case that resulted in Duronio’s conviction. The avenger was sentenced to eight years without parole in December 2006.
“If you have a disgruntled employee in a company with wide access, such as the type Mr. Duronio had, it is a high-risk combination for the company,” says Jones. “IT personnel may generally work behind the scenes and out of sight, but you have to remember the power they can hold over your company if they decide to go rogue.”
Anti-rogue defense: According to studies by Carnegie Mellon, most insider damage happens 10 days before an employee’s last day. Be sure to lock down key systems and have audit and password recovery systems in place before wielding the ax, says Ammon.
A better strategy may be to keep employees from becoming disgruntled in the first place, says Peter Hart, CEO of Rideau Recognition Solutions, which helps organizations implement rewards programs. Peer-to-peer recognition systems work particularly well for IT personnel, where even a virtual atta-boy (or girl) from a colleague can make a huge difference, he adds.
“All companies, good and bad, experience rogue behavior,” he says. “But you can really mitigate it with a good rewards system.”