You can deploy all of the firewalls and intrusion-detection devices money can buy to protect your network from hackers and malicious code, but when it comes to truly knowing what’s happening on your network, there’s no substitute for digging through system log files.
Telltale signs that appear in logs offer strong indications that you may have been hacked, say experts. The following should raise alarms in any IT administrator’s head: long entries of random characters, repeated occurrences of “..\”, passwords that have been changed by someone using a user ID of “0” and a null log-in, unexpected configuration changes to systems or devices, and a sudden increase or decrease in the number of log messages generated, to name just a few. But even with these known indicators, uncovering hacker activity can be difficult.
“Log analysis is a dirty job,” says Chris Kirschke, senior security analyst at Santa Clara, Calif.-based Silicon Valley Bank, where 1,200 users in 27 locations create 25GB of log data every month. “Logs are where you go to find out what has happened,” Kirschke says.
System logs can differ slightly from operating system to operating system, but they all follow a general format: date/time stamp, IP address of the system, the name of the process that has created the log entry and the process ID number that is assigned by the system. Knowing the format of a log entry for a legitimate event is critical to being able to identify red flags.
A system log on a Unix machine, for example, may have an entry that appears legitimate at the beginning but contains the remnants of a buffer-overflow exploit at the end, says Chris Romeo, an incident response and investigative expert at London-based Cable & Wireless PLC.
Although Web server logs aren’t as useful to intrusion analysis as system logs are, there are entries that can tip you off to hacking activity. One such entry in Web servers running Internet Information Server includes multiple occurrences of “..\ ..\.”
Jeff Rovelli, an FBI special agent in New Haven, Conn., says identifying and locating just such an entry was one of the key first steps that led to the arrest of a hacker who had used the vulnerability to gain access to a major e-commerce company’s customer database for use in a major scam. Here’s an example of this type of suspicious entry, also known as the Web traversal vulnerability: http://host/index.asp?something=..\..\..\..\WINNTsystem32\cmd.exe?/c+DIR+e:\WINNT\*.txt.
Some log entries can also show how a worm may be exploiting a back door left open by another malicious intruder. For example, in September 2001, the Nimda worm’s incessant scanning created a surge in log activity that should have been a red flag to many administrators. Nimda scanned for 16 known vulnerabilities, including the Unicode and directory traversal vulnerabilities, a total of 13 times each. That meant that every time a Nimda-infected system tried to infect another system it generated 208 log entries.
But sometimes log entries aren’t enough. For example, the Slapper Linux/SSL worm discovered in September 2002 exploited a buffer overflow in Secure Sockets Layer (SSL) Version 2. However, log entries indicating an error only show up when the worm probes an Apache server running the patched version of SSL, says Tina Bird, head of IT systems and services at Stanford University. But when the worm hits a vulnerable system, it leaves no evidence in the Apache logs or in the system logs, she says.
Other forms of malicious code, such as Trojan horses, can be lurking in your systems without showing up in log files, according to Mike Geraghty, vice president of high technology investigations at Newark, N.J.-based Prudential Financial. Geraghty says it’s important to look through the win.ini and system.ini files for suspicious executables that are being initiated, as well as through the registry for odd or unusual .exe or .dll files.
“To understand (malicious) behaviour, you have to understand what the normal behaviour is,” says Bird.
