You think your job is bad? You ain’t seen nothing until you’ve had to pick moldy food and cockroaches out of a dead PC or been asked to find out what your coworkers have been up to online when they were supposedly working. You definitely haven’t earned your IT creds until you’ve stood in two feet of water holding a plugged-in server while trying not to get electrocuted, found yourself inside a sniper’s crosshairs while you’re attempting to install a communications link, or had to worry about bombs going off while you’re futzing with network protocols.
[ Also on InfoWorld: For more dirty IT jobs, see “The 7 dirtiest jobs in IT,” “Even dirtier IT jobs: The muck stops here,” and “The dirt locker: Dirty duty on the front lines of IT” | Find out which of our eight classic IT personality types best suit your temperament by taking the InfoWorld IT personality type quiz. ]
In our fourth installment of the Dirty Jobs series, we visit with the dedicated geeks who hold jobs like these and ask how they managed to survive or, in some cases, thrive under difficult conditions.
Next time you’re hating your job, remember: It could be worse.
Dirty job No. 1: Systems sanitation engineer
Beer cans. Food wrappers. Cigarette butts. Moldy bread. Cockroaches. Things you’d typically find in the bottom of your average dumpster — only in this case, the dumpster is the shell of a discarded computer.
It’s all part of the job at Redemtech, an IT asset disposition firm that processes the aging hardware Fortune 500 companies no longer want. Somebody has to go through each piece and muck it out, decide what can be saved and what must be discarded, says Chomroeun “C-Ron” Sith, technical supervisor for Redemtech’s Grove City, Ohio, facility (and no relation to the Dark Lord).
Though it varies widely, Sith says approximately half of the systems he sees can be refurbished and resold. The other half gets recycled in an environmentally responsible way. Before that happens, they have to be inspected and cleaned — and that’s where things can get nasty.
“Some of these things look like they’ve been sitting the back of a warehouse for years,” he says. “They come in covered in dust, with cobwebs, rat droppings, and roaches inside. Sometimes they’re so rusted that when you pick them up your hands turn orange. One of the systems we got in was covered in makeup. Every time my guys touched it, they got all glittery.”
Then there was the time they opened up a desktop CPU and found a dead animal.
“It may have been a rodent or a bird,” he says. “We couldn’t tell for sure. But it was definitely dead.”
Sith estimates less than 5 percent of the 6,000 to 10,000 items his facility processes each week arrive in such bad shape they have to be wrapped in plastic to avoid infecting his staff or causing allergic reactions. Still, that’s plenty.
“Every other week some associate comes to me and says, ‘I don’t want to touch that system, I can’t take these cockroaches any more, this is ridiculous.’ I get that complaint a lot.”
Dirty job survival tip: Don’t wear your Sunday best.
“I tell my guys, ‘You can wear whatever you want, but don’t get mad if you come home dirty,'” Sith says. “That happens about 99 percent of the time.”
Dirty Job No. 2: The shadow
You probably don’t want to know where your coworkers are going on the Web. But sometimes you have no choice. Nancy Hand knows this, well, firsthand.
Until three years ago, Hand was a network engineer for a large public utility in the Southwest. When any of that site’s 3,000 employees got a malware infection, Hand received an alert via the utility’s McAfee software. She was then called in to investigate by remotely combing through the employee’s browser cache, looking for the source of the attack.
[ Get a $50 American Express gift cheque if we publish your anonymous experiences from tech’s front lines. Send your story of a lesson learned, of dealing with frustrating coworkers/end-users/managers, or a humorous happening to offtherecord@infoworld.com. ]
Along the way, Hand got to see where these employees had been surfing while they were allegedly working. Most of the time what she found was benign — a lot of sites devoted to cooking, fashion, cars, and day trading. Inevitably, though, she’d encounter the darker side.
Like the employee who swore it was a spam email that caused his browser to visit that members-only bondage site, though how that email also managed to create a member profile for him was less clear. (Hand says the head of IT security jokingly awarded her the “Golden Garter Belt” for uncovering that one.)
Or the company vice president whom Hand discovered had been spending work time visiting TeenageVirginSluts.com. Naturally, he was the VP of IT.
“He was my boss’s boss’s boss,” she says. “After I found this I contacted my manager and said, ‘We have a zero-tolerance policy for this kind of thing, so I am officially notifying you of what I found.’ To my knowledge nothing was ever done. About a year later that VP got fired for another sexually related infraction.”
Some employees ended up being escorted from the facility by armed guards, though Hand never knew whether it was due to something she had found.
“Sometimes it was a little unsettling to be on the machine of someone I’d met in another part of my job who seemed like a very toe-the-line type of person, only to discover it wasn’t true,” she said. “People aren’t as innocent as they seem. And the next time they complained about catching a virus, I’m thinking ‘Well, you sort of did this to yourself.'”
Dirty jobs survival tip: Be prepared to go it alone.
“Management didn’t seem to take any of it very seriously, even after we got hit with a denial-of-service attack that put us out of business for almost an entire day,” she says. “It all becomes political, even though you think it shouldn’t be. The VP is given a pass, but the secretary gets fired.”
Dirty job No. 3: The human server rack
The panicked call at 3 a.m. is a sad fact of life for many system administrators. But not as many admins are woken in the dead of night and asked to part the floodwaters, perform acts of impromptu structural engineering, or serve as a piece of inanimate equipment.
Brian Saunier got such a call six years ago when he was a sys admin for a small Internet service provider in Georgia. An unusually large summer storm had clogged the drain outside the ISP’s building, causing a foot of rainwater to flood the first floor, where the server closet was housed.
Fortunately, the servers were protected by an airtight glass door, says Saunier, who’s now a network administrator for Cobb Energy Management. Unfortunately, the storm also knocked out the power, causing the cooling system to shut down and putting the servers in danger of overheating.
The door had to be opened. To complicate matters, the machine containing the ISP’s customer database was sitting on the floor of the server room, directly in the flood path.
First, Saunier and two fellow sys admins constructed a dam out of cardboard, towels, and anything else they could get their hands on to keep the water out. Then Saunier was elected to run in and grab the server before the waters reached it.
“Our plan was to open the door and run in and pick up the server, which I managed to do without incident,” he recalls. “But on the way in my foot clipped the dam and the water started pouring in. I was standing in a flooded server room in two feet of water holding a powered-on server and power cords. That was disconcerting.”
After about 10 minutes, Saunier’s colleagues located a table that fit inside the closet, so he could put the machine down and commence with mop-up operations, which lasted well into the following evening.
As with all storms, there was a silver lining. Saunier submitted his story via Facebook to Ipswitch Network Management Division, which named him a “SysAdmin All-Star” for going above and beyond the call of duty. His prize: an Apple iPad, which should prove easy to hold no matter how much water is swirling around his knees.
“It was actually an entertaining experience and a great story for getting a laugh now,” he says. “Besides, there’s really no way to avoid things like this, unless you want to be in the unemployment line.”
Dirty jobs survival tip: Always pack hip waders. And make sure your server room has a raised floor — before the floodwaters start to rise.
Dirty job No. 4: Political geek
Looking for an exciting career filled with crazy hours, zero job security, bomb threats, sharpshooters, and all-nighters spent on the server room floor? You may have a future in politics — or at least, the IT side of it.
Russell Henry’s first tech job ever was working as a computer technician for a national political party during the 2000 and 2004 elections. It wasn’t your usual IT gig. Due to the cyclical nature of fundraising, staffing rose or fell with the season and offices got shuffled every few months.
“I was constantly reimaging, deploying, decommissioning, and moving PCs,” he says. “It seemed like it would never end.”
As the first Tuesday in November got closer, workdays got longer and weekends off became a vague memory. On election night 2004, Henry was given a sleeping bag and a sack of Doritos and told to camp out in the data center that housed the party’s website, just in case it needed to be manually rebooted. (It didn’t.)
Fortunately for Henry he worked for a national party, not an individual campaign, so he didn’t find himself looking for a job the day after his candidate lost. Still, doing IT in D.C. meant dealing with occasional bomb threats, anthrax scares, and other risks to life and limb.
“We had a few offices in the building next door to our headquarters,” he says. “One day we were up on the roof, installing a data laser to link the two buildings. Suddenly the Capitol Hill police burst out of the stairwell. Apparently they were a little jumpy about us pointing something that looked like a bazooka so near to federal buildings.”
Henry now does desktop support for an East Coast university, where he says life is much calmer now.
“The political job was much more stressful and intense, but also more exciting.”
Dirty jobs survival tip: Bank some cash — you may need it if your candidate loses. But don’t go into politics for the money.
“There are a lot of jobs where people are just going in for the paycheck,” Henry says. “But in politics you have to believe in the mission and be dedicated to winning. That’s a bigger motivating factor than the money.”
Dirty job No. 5: Cloud data jockey
It’s all the rage. The route to IT nirvana is to outsource your data to a cloud provider and let their servers do the heavy lifting while you kick back and enjoy life on easy street. Right?
Not exactly, says Jason Wisdom, an IT consultant who’s served as a DBA for several clients and discovered there’s a darker side to every silver-lined cloud. When you’re administering data that lives in the cloud, your troubles are only just beginning.
In the cloud you’re still responsible for data integrity and security, but with almost no control over the hardware where it lives. Performance slowed to a crawl? System crashes? Failed backups? It may not be your fault, but it is your problem.
“With one of my clients their entire cloud farm just collapsed due to a failed motherboard,” he says. “The disk array was ruined as well. All the servers, including the database, had to rebuilt from scratch. The entire IT infrastructure was down for a week as servers were reinstalled and the data was being recovered.”
Fortunately, says Wisdom, the company has backed up the data to a stand-alone machine, or it really would have been in trouble.
There are other clouds on the data horizon. Portability of data between different systems, legal issues about where sensitive data can reside, even just knowing that data has been “destroyed” and is not still sitting on some hard drive somewhere — all of these grow far more complex in the cloud, which Wisdom says most service providers are loathe to admit.
“When the cloud is a managed third-party service, the third party now has control of the data, not the DBA,” he says. “There is no way to guarantee safety of that data from prying eyes, malicious intent, or accidental, third-party-caused data loss. Even in a local cloud environment, there are still more hands that can get to highly sensitive data, which is one of the DBA’s most important concerns.”
Dirty jobs survival tip: The cloud is great for many things, but database management isn’t necessarily one of them. Keep your data close and your DBAs closer.
Dirty job No. 6: Network infantryman
Network engineers may have the least glamorous job in all of IT. They get to squeeze through dusty attics and muddy crawlspaces, pulling cable and fixing gear. If you’re doing this dirty job for the military, there’s a bonus — you also get to deal with bombs exploding overhead and bullets whizzing by.
As a network engineer for the U.S. Air Force, Josh Stephens spent four years designing, deploying, and managing large-scale classified and unclassified networks. The dirty part of that job included climbing into flooded manholes to troubleshoot malfunctioning network equipment, setting up tent-based mobile command posts during torrential rainstorms, and digging out mud-mired Humvees serving mobile comm links.
Though Stephens says he was never deployed in a war zone, as “head geek” for IT management software vendor SolarWinds, he gets a lot of calls from network guys in the line of fire.
“When you’ve got to get in there and get the network running in a forward command post, you’re on the front lines,” he says. “When the United States took over Iraq, we got calls from customers who were inside Saddam’s palace and needed help setting up the network. They were still very much in the hot zone at that time.”
Stephens says he spends about a third of his workday talking with military guys who need networking help and don’t have time to go through official channels.
“When you’re in the field getting shot at, you get the tools you need to get the job done,” he says. “I’m a VP at SolarWinds, but if some guy in Iraq calls me at 2 a.m. because he has a problem getting his network set up, I get up and take the call.”
Dirty jobs survival tip: Suck it up and do it. But be sure to keep your head low.