Ten years ago, Michael Riva was network administrator for a top-five American consultancy. Employees were downloading graphic pictures and videos onto the network. Riva told his boss a proxy server with content filtering might be in order; his boss laughed and suggested they put in a bigger file server instead.
Riva stood his ground and was gone a few weeks later.
Though some would look at this as a career mistake on his part, Riva would gladly do it again. What’s right is right, he says.
After asking security professionals about some of the worst (publishable) mistakes they’ve ever made on the job, CSO decided to ask about mistakes professionals would gladly make again. Best mistakes, if you will, that ultimately paid off either personally or professionally.
Here are five such tales:
1. THE PORN DISCOVERY
Mistake maker: Michael Riva
Position: Network administrator, IT consultant
Location: Currently in Ireland
The incident: Bosses didn’t like his idea of blocking porn
“Ten years ago I was working for one of the top-five American consultancy companies as a network administrator. At the time we had a file server with 40GB of disk space. This was pretty descent space, knowing that it was serving over 500 people. Over a short period of time, however, the disk space went to less than 500 MB. People had problems saving documents and so on.
“One day I decided to clean up a bit and implement disk quota with a third-party tool for the Windows NT 4 server. To my surprise, I noticed more [network space] being used by some consultants who saved a lot of pornographic content. There were all sorts of crazy things on there. I deleted the content without asking anything and a few days later the content was back. A few people were actually flooding the server with those videos.
“I spoke to the IT manager and suggested we implement a proxy server with content filtering. He laughed at me and told me we would buy a bigger file server instead. He also said the consultants are the fuel of the company; each one of them was bringing [US]$1,500 to the company every single day.
“After a few weeks I was gone because the senior partners heard about this and my quest to eradicate those files did not please them. I reckon that in such a situation nowadays the IT manager would have been let go instead. I was 22 years old at the time but was still shocked by the behavior.”
2. STICKING A SOX IN IT
Mistake maker: Anonymous
Position: Director of information security
Location: Northwestern U.S.
The incident: Botched initial Sarbanes-Oxley (SOX) compliance initiative
“In year one of SOX, we had outlined a compliance plan and set of controls that we thought would meet the requirements. Of course, in year one, every company was making educated guesses about what their auditors would require. Because of the timing of our fiscal year, we were among the first companies to have to comply, so we had very little concrete information from other companies.
“When our auditors showed up several months before year’s end, they delivered some very bad news: We had gotten it all wrong. Our controls would not meet their minimum requirements. We had to start over with only a few months to implement and test our entire SOX compliance plan and most of our controls. This was a task that had taken many companies two years to complete.
“We dropped everything, rallied the entire organization, and put in place an extremely lean and effective set of SOX controls. Not only did we pass, our auditors later said that our program was among the best and most efficient they had seen.
“While other companies built massive and complex SOX compliance programs that they would spend the next few years trying to pare down, we had built a minimalist program from the beginning out of absolute necessity. We simply did not have time to over-engineer the SOX program, so we focused on meeting the minimum requirements.
“Our error in the initial design of our SOX program and subsequent focused and hurried redesign resulted in an industry-leading program that was later praised by peers at much larger companies who had spent many times more on their programs.”
3. CRASHING THE FIREWALL
Mistake maker: Christine Wanta
Position: CSO at IntraISP, a Clearwire company
Location: Greater St. Louis area
The incident: Suffered a firewall crash and had to rebuild rulebase from scratch
“I had a firewall rulebase crash on me while I was in it, and since I was the one in the rulebase I had to fix the situation. We did have a strong backup procedure prior to any changes, but I either did not create the backup or screwed it up or was being taught a lesson from one of my fellow admins. I don’t recall.
“I had to rebuild the rulebase from scratch basically using a not-so-friendly output of a backup, going through the rulebase change requests and basically cleaning the rulebase up as I went.
“I learned more about that particular network and our rules structure than I would have just working on them.
“Everyone in security makes mistakes. It is the one true way we learn to succeed in our field. Constantly testing and pushing the limits will break something. I prefer to be the breaker.”
4. KIDNAPPING THE WRONG GUY
Mistake maker: Douglas McGaughey
Position: Security consultant
Location: Washington D.C.
The incident: Kidnapped the wrong person during a counter-terrorism operation
“I used to manage a counter-terrorism training team and one time, after conducting a lengthy target evaluation and feeling confident in our plan, we initiated it. We kidnapped the individual, who liked to take early-morning jogs. We hustled him into our waiting van and then identified ourselves. “Long story short: It was the wrong guy. We came to find out that our intended victim had an overnight guest up from some assignment in Central America.
“But we came out smelling like a rose. The guy we kidnapped was so impressed with our stealth, speed and capabilities, and disappointed with himself for letting his guard down so quickly after leaving his last “high-paced” assignment that he offered to give a speech to our students [about his experiences]. His speech hit home and we lucked out big time.
“I wouldn’t change anything, as it taught us some valuable lessons and the audience learned something.”
5. RISKING THE JOB FOR WHAT’S RIGHT
Mistake maker: Doc Farmer
Position: Senior security specialist Location: Virginia
The incident: Losing jobs for taking a stand
“I’ve lost a job or two by sticking to my principles and telling the truth to management who didn’t want to hear the truth (not even diplomatically). When they’ve taken risks that put the entire organization at risk and asked me to sign off on them, I’ve refused. And I’ve been sent packing because of it.
“But even with that, I don’t regret the decision. I have to be able to look myself in the mirror when I shave my head every morning, after all, and I’d rather be unemployed and honorable than being an employed liar.”