Linux PCs sending “ICMP udp port unreachable” to DNS server

domain-name-systemicmplinuxnetworkingreverse-dns

So digging around in my syslog I have been noticing a lot of ICMP packets being flagged by our ASAs:

 %ASA-4-313005: No matching connection for ICMP error message: icmp src Internet:x.x.21.122 dst MGMT:x.x.36.55 (type 3, code 3) on Internet interface.  Original IP payload: udp src x.x.36.55/53 dst x.x.21.122/47927.

Doing a tcpdump on the originator ( x.x.x.122 a Linux machine) of the ICMP reply I notice that a DNS query request is sent, and after some time, the DNS server replies. Immediately upon the reply the Linux server sends a message that the port is unreachable back to the DNS server. Please see below:

19:29:06.684523 IP x.x.21.122.47927 > DNS.domain: 7182+ PTR? x.x.x.x.in-addr.arpa. (43)
19:29:11.690336 IP x.x.21.122.33897 > DNS.domain: 58231+ PTR? x.x.x.x.in-addr.arpa. (45)
19:29:13.850887 IP DNS.domain > x.x.21.122.47927: 7182 ServFail 0/0/0 (43)
19:29:13.850929 IP x.x.21.122 > DNS: ICMP x.x.21.122 udp port 47927 unreachable, length 79

19:29:16.692581 IP x.x.21.122.33897 > DNS.domain: 58231+ PTR? x.x.x.x.in-addr.arpa. (45)
19:29:21.697217 IP x.x.21.122.42976 > DNS.domain: 19120+ PTR? x.x.x.x.in-addr.arpa. (45)
19:29:22.977289 IP DNS.domain > x.x.21.122.42976: 19120 ServFail 0/0/1 (56)
19:29:22.977342 IP DNS.domain > x.x.21.122.33897: 58231 ServFail 0/0/0 (45)
19:29:22.977382 IP x.x.21.122 > DNS: ICMP x.x.21.122 udp port 33897 unreachable, length 81

I thought that maybe the Linux machine was just not waiting long enough for the DNS reply, so I increased the wait time in /etc/resolv.conf… But with no luck.

I understand that the Linux PC is doing a reverse DNS lookup, and the DNS server is replying that it cannot resolve the name (because it does not exist in the DNS server. The particular hosts it is querying for do not have an entry on the DNS for a reason). This is why the request is taking so long. But I just want to find out how to modify the Linux PC so that it does not send out these ICMP messages each time this happens.

If anyone could please help me figure out how to stop these messages from showing up it would be much appreciated.

ASA configs with show run | inc icmp below

 ASA# show run | inc icmp
    icmp unreachable rate-limit 1 burst-size 1
    icmp permit any echo Internet
    icmp permit any echo-reply Internet
    icmp permit any echo DATA
    icmp permit any echo-reply DATA
    timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

Best Answer

That sounds familiar, and I've seen the same behaviour coming from Linux based services.

It's an application-specific behaviour, although I can't recall which application I traced this too.... or even if I did actually trace it to an application...

Basically, the application in question does its own DNS lookups using the normal libC resolver (which is normal in the case where something like nscd or dnsmasqd is not running as a local cache), and the client socket is closed before the response gets back. Because the response comes back and there is nothing listening on that (unconnected UDP) socket any more, it responds with a port unreachable.

Two things can be improved:

1) use a client side DNS cache (preferably not nscd if you care about short TTLs)

2) (untested) drop on the OUTPUT chain ICMP-unreachable packets going to port UDP/53

I would suggest the former to be the preferable solution, and this probably accounts for why you don't tend to see this coming from Windows machines (which have a local cache)

Related Topic