Last year the CERT Coordination Center, a repository of information technology vulnerability reporting run by Carnegie Mellon University, posted 3,784 vulnerabilities. Admittedly, it was a slight dip from the 4,129 reported in 2002, but nonetheless it points to a growing trend in software fallibility. In fact, prior to 2000, when 1,090 vulnerabilities were reported, the total never hit 500.
What this means to the millions of IT security personnel is doubly troubling. First, software, despite all the claims, isn’t getting any more secure. And secondly, the problem of patching vulnerabilities has moved from an afterthought to a full-blown crisis. Throw into the fray an increased level of malicious code activity targeting not only software vulnerabilities but also end user ignorance and the picture painted, of teetering IT security, is not pretty. But before you unplug your machines from the Internet, consider this: although the situation is unlikely to dramatically improve in the immediate future, better tools and technology to administer patches will make the scourge more manageable.
And, of course, most IT managers don’t have to patch thousands of vulnerabilities each year – maybe just a few hundred.
When Ariel Silverstone, chief information security officer at Philadelphia-based Temple University, was asked if he had to patch seven or eight hundred vulnerabilities a year, he said, “I estimate it to be much more than that.” Even though the university has pretty much every operating system under the sun, 75 per cent of the systems are Microsoft and about 20 per cent are Mac. “We are way past frustration…it is part of life and we have to deal with this,” he added.
And deal with it he does. “First of all you need to know what you are doing,” Silverstone said – and he was not being flippant. “Are you addressing only security patches or all the functionality patches that Microsoft has?” An example of the latter is the patch for Office 2003, designed to get rind of the swastika character in a certain font. “Should it be driven by security or by the help desk?”
This problem, one that is found in many companies, occurs when there is no corporate mandate around patching.
Patching “has to be policy driven…and come from the top,” said Victor Keong, partner, security services with Deloitte & Touche in Toronto. “Mature organizations will have CSOs who drive that process, otherwise they rely on the CIO or director of IT.” To be successful you “have to attack it from bottom up…and the top down,” he explained. Bottom up includes choosing the right patching tools and doing the actual vulnerability assessment. Top down deals with corporate policies on how to react to various vulnerability scenarios. “They need to go hand in hand…(and) if you (are coming from) both angles, then I think you are closer to the Holy Grail.”
The Toronto Catholic School Board has a centralized approach to policy and a decentralized approach to implementation. Dave Klein, a network analyst with the board, is part of a team that decides which patches to implement and the deadlines for them to be done. The group usually gives server owners (each owner typically has less than half-a-dozen servers to patch) two or three days to patch a vulnerability, he said. But deadlines may vary depending on the actual applications running on a server and the level of the vulnerability associated with the patch. “Obviously if there is a situation like a Blaster or a Welchia going through the system, that really comes to the top,” Klein said, and we “will start pushing out patches immediately. In this case “we may have to accept that a certain percentage of patches may not go through, or may leave a system that itself will have to be reimaged later if the patch fails.”
FINDING, TESTING, INSTALLING
Prior to any action being taken, a company first has to be aware of any new vulnerability as it comes down the pipe. Luckily this is the easiest of the three aforementioned aspects of patching. But there is a caveat: it is easy only if you know what you have.
Tom Slodichak, chief security officer with WhiteHat Inc. said he dealt with one customer who was constantly dealing with an ever-changing network. “They have a lot of machines that connect to the network daily, that aren’t part of the overall domain,” he explained. “Despite the greatest efforts 20 to 25 per cent of the machines that inhabit his network any one week are not on his antivirus push list.”
To solve this sort of conundrum Temple’s Silverstone controls the network access centrally. His rule is simple: buy whatever machines you want, “but if they don’t fit (meet certain requirements) they will never get connected to the network,” he said. If a group bought a Windows 3.1 machine, “we just wouldn’t let it go on the network.” This centralized approach allows for greater understanding of who has what, and what vulnerabilities are potentially out there.
The technology to keep IT apprised of vulnerabilities is mature; there are free patch management tools available from the likes of Microsoft (software update service or SUS) to the more expensive, subscription-based, models from companies like Symantec Corp. (DeepSight). Basically these tools tell you what you have that is vulnerable, and often send the requisite patch to you. But the tools are often platform specific.
“The first thing that any decent patch management tool will do is to certify what you need and whether or not you have it,” Silverstone said. SUS is much improved, Silverstone said, but “in our case a lot of our systems are not Microsoft, so solving 75 per cent of the realm is good but it is not the entire problem…so pretty much to do that you need to go with a full blown patch management solution.” These are also typically subscription-based and not cheap, Silverstone said. He added that Temple is looking into a solution that would fully manage patching (except the testing portion: “I am a doubting Tom,” about another company’s ability to properly test patches, he said). “Then we click to approve or disallow the patches to be applied,” The cost is about US$100,000 per year.
A patch management solution also needs the infrastructure in place to support it. For example, an organization needs to decide how many servers will run the system, who will administer them and what level of bandwidth can be sued. “It is not something that you put on a desktop and go away,” Silverstone added.
There are also dozens of security bulletins to keep IT apprised of every vulnerability in existence. The vendor specific sites have the advantage of more detailed information, but the disadvantage of being only one stop of many. Third-party sites are a great starting point since they are vendor agnostic – but finding exactly what affects your network can be more of a challenge.
“In the (monthly security) bulletins we offer mitigation advice and work-arounds, and tell people other things they can do if they can’t deploy the patch,” said Amy Carroll, director of product management of the security business and technology unit with Microsoft in Redmond, Wash.
“We have also found that Microsoft is getting a lot more proactive in terms of contacting their clients…so that definitely is helping,” Klein said. Regardless how a company decides to approach the identification of vulnerabilities and implementation of patches (doing it themselves or relying on third-party software), the testing portion of patching pretty much has to be done in-house. Even Microsoft (which tests patches against a variety of third-party software) admits it is next to impossible to guarantee a patch’s success on a specific system given the almost limitless combination and configurations that can be deployed.
DID YOU TEST THAT?
In an ideal world, every patch would be tested for weeks and put through a worse-case-scenario ringer that would make NORAD proud. But the simple fact is most IT patching is ruled by the clock and the wallet.
The Toronto Catholic School Board does not have a dedicated team to test patches, so often Klein and his cohorts wait to see if other organizations are having a problem with a patch before installing it. “One of the problems we have is that we don’t have a huge number of machines on which to do testing,” he said. For example, they don’t have a test system for their SAP financial system so patches are often installed untested. This by no means implies that it is done automatically or without forethought.
“A lot of the responsibility is actually delegated to the server owners,” Klein said. So the financial group (which runs the SAP servers) “will go through their own review process to see if A, it is critical and B, talk to SAP to see if (there are) any implications to applying the patch,” he said. “Obviously the last thing we want to do is take our SAP system down because of a patch. If [the patch] is absolutely critical we make sure there is a full back up system before we apply it and leave ourselves a back out option in case something goes wrong,” Klein said. “We would not proceed with a patch unless we know perfectly well that we can recover,”
It takes Silverstone and his team about a day to test a new patch. But since it happens, on average, “more than twice a day,” it is time consuming. Temple is looking into creating a dedicated team for patching but “admittedly that would require a full-blown administrator,” he said. Temple is also looking into investing in technology that would allow it to test all the machines on the network to see if their patches are up-to-date, and if not, refuse the machine access. “In the case of antivirus, we absolutely do it,” Silverstone explained.
INSTALLING PATCHES
The technology that exists today, although far from perfect, does allow systems administrators the ability to automatically update machines. For places like Temple with 33,000 computers and the Toronto Catholic School Board with more than 16,000, this automation is critical.
But Silverstone says it is not as easy as it seems. You need to get people used to the idea that somebody is going to be looking at the software on their machine, he said, “so it is a political issue.” Admittedly, in a corporate environment, where users have few rights, this is less of a problem. But still, when reboots are necessary, both Klein and Silverstone realize bowing to the calendar of the end user is often a necessity.
However the patching game plays itself out in the years to come, most software vendors are committed to creating less vulnerable software. Even Microsoft, often accused of being insensitive to the plight of the patcher, has started its road to redemption. Steve Ballmer, its effervescent CEO, recently told ComputerWorld Canada when he was in Toronto, that Windows Server 2003 had nine critical vulnerabilities in the first 10 months versus 40 for Windows Server 2000.
“Perfect, no. Big improvement, absolutely.”
