Question for someone who knows a lot more about networking than I do…
So, long story shortish, I had an issue with a remote server tonight that I couldn't get corrected over ILO (HP's out-of-band management technology), even after doing a complete RTFM. I noticed that the ILO firmware version on this server was over 5 years old, and figured that might be the cause of my difficulties, so I went to upgrade it, and couldn't. (The ILO web management console provides a mechanism to do firmware updates.) In a mistake I'll regret for a long time, I decided I could call tech support and ask how I could get the firmware updated. Upon actually listening to their advice and doing a soft reset of the interface (also through the ILO web management console), it became non-responsive.
Expected behavior, and I started a ping -t to the ip address of the ILO interface so I'd know when it came back up. Except that the pings were all coming back as Reply from 0.0.0.0: ... I decided to give it a few minutes anyway, in the event that the reset was doing something weird. Well, half an hour later, it was still replying from 0.0.0.0, so I had someone unplug the server. This changed the ping replies to Request timed out. As soon as it was plugged back in, the ping responses from 0.0.0.0 started up again. Even now, 6 hours later, a ping to the "actual" IP address of that ILO interface responds with a reply from 0.0.0.0.
For what it's worth, we do not have a separate VLAN or subnet for our ILO (I'm working on it), and pinging addresses in the subnet with nothing on them does return a Request timed out. A tracert to the address goes through our switches, out the link at our main office, to the other side of T1 at the remote site… and then shows 0.0.0.0. A ping request from over the remote VPN (which is a separate VLAN and subnet) comes back Request timed out.
From everything I know about networking, which is apparently less than I thought I knew as recently as yesterday, it's not possible to get a reply from 0.0.0.0. That's the "any" address. So even if the ILO interface had its address reset to 0.0.0.0, it shouldn't be able to send a reply as 0.0.0.0, so… well, does anyone know why this is happening? Or how this even could happen, for that matter?
UPDATE: Got someone on site to check the settings on ILO, and sure enough, the IP address of the interface got reset to 0.0.0.0.
Best Answer
This is most likely a firmware bug, but the only situation I've heard of that -sources- packets from 0.0.0.0 is RARP (reverse ARP) which sends out requests from an all zero MAC to find its IP address. This used to be a component of diskless boot once upon a time but hasn't been seen in quite a while.
It may be that the pinging station still held the ILO's MAC address in cache - and, of course, the intermediary switch. If the ILO had lost its address it could have been going through a RARP / BOOTP process. Or it just lost its mind.