Netstat and ss Not Seeing Connection – Troubleshooting

kubernetesnetstatnetwork-namespacetcpdump

I'm trying to find which application running on Kubernetes cluster is using a specific port to communicate with RabbitMQ cluster. On RabbitMQ I see that connection is coming from 192.168.1.10:34226. 192.168.1.10 is one of Kubernetes nodes. After logging in to 192.168.1.10 I checked if connection is established from that node by tcpdump command tcpdump -i eth0 port 34226. I see traffic with both directions. Now I'm trying to find which process is making this connection. First I tried netstat:

netstat -tapn | grep 34226

and it found nothing. Then I tried ss but still nothing. Connection is established, with tcpdump I can see heartbeat packets sent to RabbitMQ and response from RabbitMQ. But netstat and ss don't report established connection.

UPDATE 1

I found that is something related to network namespaces. My question is similar to this question but I need to find PID of process which make connection from 34226 port.

Best Answer

After couple of weeks I found answer. I needed to run netstat -tapn in all namespaces with following command:

sudo ip -all netns exec netstat -tapn | grep 34226

Now I have PID number of process which start tcp connection from 34226 port.

Related Solutions

Lsof not showing what port a proc is listening on

You'd rather use pfiles which is part of Solaris and supported by Sun, unlike lsof.

Its usage is slightly different as it expects a pid as argument but you can achieve what you want with something like:

pfiles $(pgrep libhttpd)

or even, if you don't know the process name:

pfiles /proc/*

How to make cURL use keepalive from the command line

curl already uses keepalive by default.

As an example:

curl -v http://www.google.com http://www.google.com

Produces the following:

* About to connect() to www.google.com port 80 (#0)
*   Trying 74.125.39.99... connected
* Connected to www.google.com (74.125.39.99) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: www.google.com
> Accept: */*
> 
< HTTP/1.1 302 Found
< Location: http://www.google.ch/
< Cache-Control: private
< Content-Type: text/html; charset=UTF-8
< Set-Cookie: PREF=ID=0dd153a227433b2f:FF=0:TM=1289232886:LM=1289232886:S=VoXSLP8XWvjzNcFj; expires=Wed, 07-Nov-2012 16:14:46 GMT; path=/; domain=.google.com
< Set-Cookie: NID=40=sOJuv6mxhQgqXkVEOzBwpUFU3YLPQYf4HRcySE1veCBV5cPtP3OiLPKqvRxL10VLiFETGz7cu25pD_EoUq1f_CkNwOna-xRcFFsCokiFqIbGPrb6DmUO7XhcpMYOt3dB; expires=Tue, 10-May-2011 16:14:46 GMT; path=/; domain=.google.com; HttpOnly
< Date: Mon, 08 Nov 2010 16:14:46 GMT
< Server: gws
< Content-Length: 218
< X-XSS-Protection: 1; mode=block
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.ch/">here</A>.
</BODY></HTML>
* Connection #0 to host www.google.com left intact
* Re-using existing connection! (#0) with host www.google.com
* Connected to www.google.com (74.125.39.99) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: www.google.com
> Accept: */*
> 
< HTTP/1.1 302 Found
< Location: http://www.google.ch/
< Cache-Control: private
< Content-Type: text/html; charset=UTF-8
< Set-Cookie: PREF=ID=8b531815cdfef717:FF=0:TM=1289232886:LM=1289232886:S=ifbAe1QBX915QGHr; expires=Wed, 07-Nov-2012 16:14:46 GMT; path=/; domain=.google.com
< Set-Cookie: NID=40=Rk86FyMCV3LzorQ1Ph8g1TV3f-h41NA-9fP6l7G-441pLEiciG9k8L4faOGC0VI6a8RafpukiDvaNvJqy8wExED9-Irzs7VdUQYwI8bCF2Kc2ivskb6KDRDkWzMxW_xG; expires=Tue, 10-May-2011 16:14:46 GMT; path=/; domain=.google.com; HttpOnly
< Date: Mon, 08 Nov 2010 16:14:46 GMT
< Server: gws
< Content-Length: 218
< X-XSS-Protection: 1; mode=block
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.ch/">here</A>.
</BODY></HTML>
* Connection #0 to host www.google.com left intact
* Closing connection #0

This snippet:

* Connection #0 to host www.google.com left intact
* Re-using existing connection! (#0) with host www.google.com

Indicates it re-used the same connection.

Use the same "curl -v http://my.server/url1 http://my.server/url2" invocation against your server and check that you see the same message.

Consider using tcpdump instead of netstat to see how the packets are handled. netstat will only give you a momentary glimpse of what's happening, whereas with tcpdump you'll see every single packet involved. Another option is Wireshark.

Best Answer

Related Solutions

Lsof not showing what port a proc is listening on

How to make cURL use keepalive from the command line

Related Topic