Two members of my security team gave their two-weeks’ notice this week. Each departed for different reasons. Our most technical security engineer left to start his own company. Our security auditor jumped ship for one of the big consulting companies. It offered him a significant salary increase, more vacation time and the opportunity to build his own team.
We tried to keep both of them, without success. In my experience with the information security business, the average tenure for a security engineer seems to be about a year. Even in today’s uncertain economic climate, keeping security professionals is a problem. But the more immediate question is, How do I keep things going until I can hire and train replacements?
With two staffers leaving at the same time, I have to make changes to the way we’ve been managing our security infrastructure. My role is expanded until we can find qualified replacements. I know that it will probably take at least a month to find people. And once we find and hire qualified candidates, it will take another month to train them and bring them up to speed. For at least two months, then, I must take responsibility for the infrastructure and programs that the workers were managing or try to delegate them.
The department’s core deployments include an intrusion-detection system, a two-factor authentication system and firewall management. In addition, we still have architecture reviews, audits, secure server baseline construction, policy creation and review, and plenty of other one-off issues that typically plague the department.
In order to deal with the increased workload, I need to off-load certain aspects of our daily activities to other areas of the company. This week, I focused on our intrusion-detection system deployment and our SecurID authentication infrastructure. The security department is responsible for incident response related to alerts generated by Tripwire, from Portland, Ore.-based Tripwire Inc. We are also responsible for the day-to-day administrative activities related to the SecurID infrastructure.
We run Tripwire on more than 50 Unix servers within our DMZ and corporate network. They are currently configured to send alerts via e-mail to an alias account to which the security department and the Unix administrator are subscribers. I also configured Tripwire to run a check on five especially critical files at 10-minute intervals. For example, the /etc/passwd and the /etc/shadow files contain information on accounts and passwords. If someone were to add an account, the password and shadow files would change, and Tripwire would issue an alert on that change. If Tripwire detects a change in any of the critical files I’ve identified, it sends an alert to my cell phone.
I don’t receive a significant amount of alert traffic, but in order to concentrate on other aspects of our security infrastructure, I need to off-load the incident response to another trusted area of the organization.
So I met with the manager of my company’s network operations center (NOC) and explained the situation. We agreed to have the NOC analysts help out by receiving and responding to Tripwire alerts. I will also configure Tripwire to send Simple Network Management Protocol trap alerts to our central enterprise management application.
We’re using San Francisco-based Micromuse Inc.’s Netcool for monitoring the status of our infrastructure. I’m writing an escalation guide and will provide the NOC teams with a short PowerPoint slide presentation to help them understand what Tripwire is and how to respond to alerts. Netcool will provide an immediate notification of the alert, and the corresponding e-mail will provide details of the alert (which files changed, when and how). The challenge will be training the analysts to determine whether an alert is legitimate.
The NOC manager also agreed to help me deal with some of the administrative issues regarding our SecurID token infrastructure. We use Bedford, Mass.-based RSA Security Inc.’s ACE server for our two-factor authentication needs. Ninety percent of the support issues involve users who leave their tokens at home, users who forget the passcodes associated with their tokens or token malfunctions that require issuance of new tokens.
RSA’s ACE implementation includes several Web-based tools to deal with these issues. One of these tools, called Quick Admin, gives administrators a predefined set of actions for token administration. The tool installs on a Web server, and administrators access it by way of a SecurID token-protected Web page. As security manager, I control who gets access to this Web tool. The administrators can execute only certain functions, but I maintain full control of the SecurID infrastructure.
Again, I created an administration guide, provided a PowerPoint slide presentation for the NOC analysts and trained them on how to deal with the main SecurID issues.
My next task is deciding how to deal with our intrusion-detection infrastructure. For now, I’m going to retune our network sensors to filter out many of the false positives and focus on a select group of signatures that I feel are important, given my company’s network, systems architecture and configuration. Once we’re fully staffed again, I’ll consider relaxing the filters, because even false-positive traffic can be useful for data correlation.
I don’t currently have the time or resources to analyze and correlate traffic. Hopefully, these actions will buy me some time until I can get new staffers on board.
This week’s journal is written by a real security manager, “Mathias Thurman,” whose name and employer have been disguised for obvious reasons. Contact him at mathias_thurman@yahoo.com, or join the discussion in our forum.