The case of the impossible IP address

This month, I found myself chasing a mystery on my company’s network.

When I get free time (which isn’t often), I try to review the logs of our various security devices. We have other people who can dig through this sort of thing, but I think it’s important to be connected to what’s going on in our network. Maybe my eyes will see something that the software tools are missing or that other people have overlooked. In this case, I validated that idea.

As I was reading through our firewall’s logs, which is usually a tedious and unexciting (though necessary) activity, I found something strange. Our firewall was seeing a lot of traffic coming in from the Internet with the destination IP address of 0.0.0.0 — an impossible address.

I long ago set up a rule on our firewall to block traffic either coming from or going to that and other clearly bogus numbers. Our firewall was dutifully blocking this bad traffic and noting that in its logs. But when I saw the log entries, I was intrigued — my boring duty had suddenly become interesting. What could be trying to send traffic into my network with that crazy, nonexistent address, and how could it possibly have been delivered to us? That’s like finding a letter in your mailbox with no address on it.

I put on my detective hat. There weren’t too many possibilities within the realm of credibility. If somebody was sending traffic from somewhere on the Internet to a destination address of all zeroes, it would go exactly nowhere. Yet here it was at my network’s door. Was there a way the address could have changed somehow once it got into my perimeter?

I didn’t think that was very likely but noted it as a possibility. But if it wasn’t coming from the Internet, the only other reasonable explanation I could think of was that the traffic was somehow being injected into our network between our firewall and the Internet, which would mean — cue the spooky music — that the call was coming from inside the house!

Naturally, it’s not easy to track down the source of traffic when the address doesn’t make any sense. I enlisted the help of my company’s network engineer, a very sharp guy. He was as intrigued as I was, so together we set out to try to figure out what was going on. We took a look at the Internet router, which is the next hop for network traffic outside our firewall. We didn’t find anything strange in the router’s configuration or traffic logs, but something was definitely fishy. There was no trace of any traffic with that strange, all-zero IP address.

After a closer look, the network engineer discovered a significant clue: The router was out of memory. Apparently, our Internet router had been working hard, and it ended up without enough free memory. In the end, that memory shortage turned out to be the culprit. This brand of router ends up “dropping” some network information when it doesn’t have enough memory, and that explains the zeroes: Without enough memory to put together a valid network packet, the result was a bunch of zeroes, some of which turned out to be in the right place to produce the mysterious empty address.

A reboot of the router seems to have solved the problem. Now our network team is going to keep a closer eye on the resource usage in our routers and network devices.