Squid Proxy timing out on Public Interface – Packets getting lost on port 3128

centos6linodePROXYsquidtimeout

Partial Solution

Turns out that traffic on port 3128 was getting lost on the way to my VPS. I am not sure if this is Linode blocking this port, or if it is something in the between. (I have used this same proxy on other clouds, and it worked.)

Changed the port to 53128, and it worked.

However, how can I probe port 3128 to check where the packages are getting dropped? It sure is before they reach my VPS.


Problem

I have a squid proxy setup in a VPS running CentOS 6.7 in order to allow me to share web sessions across multiple cluster nodes.

This proxy is working in the cloud, and when all nodes are inside the same private network 192.168.0.0/24 everything works just fine.

This week I have deployed a few servers at my home to do some very long batch jobs, and I need to connect through my proxy. However, squid is timing out on my public interface.

My squid.conf is pretty much allowing every incoming connection, as I am being restrictive trough iptables. However, even with the firewall stopped, I can't connect to my proxy over the Internet.

Test connection from VPS cluster

Note: Public IP and hostname intentionally omitted.

$ curl --proxy PUBLIC_HOSTNAME:3128 -v google.com.br
* About to connect() to proxy PUBLIC_HOSTNAME port 3128 (#0)
*   Trying PUBLIC_IP... connected
* Connected to PUBLIC_HOSTNAME (PUBLIC_IP) port 3128 (#0)
[ OUTPUT OMITTED ]
* Connection #0 to host PUBLIC_HOSTNAME left intact
* Closing connection #0

$ curl --proxy PRIVATE_HOSTNAME:3128 -v google.com.br
* About to connect() to proxy PRIVATE_HOSTNAME port 3128 (#0)
*   Trying PRIVATE_IP... connected
* Connected to PRIVATE_HOSTNAME (PRIVATE_IP) port 3128 (#0)
[ OUTPUT OMITTED ]
* Connection #0 to host PRIVATE_HOSTNAME left intact
* Closing connection #0

Test connection from home (Internet)

Note: Public IP and hostname intentionally omitted.

Packets on port 3128 do time out. Packets on port 53128 do work.

$ curl --proxy HOSTNAME:3128 -v google.com.br
* About to connect() to proxy HOSTNAME port 3128 (#0)
*   Trying PUBLIC_IP... Connection timed out
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

$ curl --proxy PUBLIC_HOSTNAME:53128 -v google.com.br
* About to connect() to proxy PUBLIC_HOSTNAME port 53128 (#0)
*   Trying PUBLIC_IP... connected
* Connected to PUBLIC_HOSTNAME (PUBLIC_IP) port 53128 (#0)
[ OUTPUT OMITTED ]
* Connection #0 to host PUBLIC_HOSTNAME left intact
* Closing connection #0

On a side note, as you can see from the above outputs, even from inside the cloud a connection CAN BE made through the public interface.

My VPS is hosted on Linode, theirs private network are made through a virtual interface, all traffic is routed through the public interface. Anyways, I don't think this is the issue.

squid.conf

Squid is listening on its default port.

$ sudo netstat -ntlp | grep  3128
tcp        0      0 :::53128                    :::*                        LISTEN      19282/(squid)       
tcp        0      0 :::3128                     :::*                        LISTEN      19282/(squid) 

My configuration file is pretty standard, with addition to a cloak at the end.

Squid was installed through yum, this is the package available for the CentOS 6.7.

$ sudo cat /etc/squid/squid.conf
#
# Recommended minimum configuration:
#
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
#acl localnet src 10.0.0.0/8    # RFC1918 possible internal network
#acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
#acl localnet src 192.168.0.0/16        # RFC1918 possible internal network
#acl localnet src fc00::/7       # RFC 4193 local private network range
#acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

visible_hostname cluster.proxy
dns_nameservers 8.8.8.8 8.8.4.4
#hosts_file none
hosts_file /etc/hosts

# quick_abort_min 0 KB
# quick_abort_max 0 KB
strip_query_terms off 
log_icp_queries off
client_db off
buffered_logs on
# half_closed_clients off

connect_timeout 30 seconds
forward_timeout 60 seconds
request_timeout 60 seconds

dns_timeout     30 seconds
# positive_dns_ttl 8 hours
# negative_dns_ttl 30 seconds

acl localnet src all           # Intentionally left open. Not sure if this is valid.
acl ghome src OMITTED.ddns.net # Dynamic DNS for my home address

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager

# Deny requests to certain unsafe ports
#http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
#http_access deny CONNECT !SSL_ports

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost
http_access allow ghome         # Tried this rule for my home

# And finally deny all other access to this proxy
#http_access deny all
http_access allow all           # Tried this rule for world

# Squid normally listens to port 3128
http_port 3128
http_port 53128

# We recommend you to use at least the following line.
hierarchy_stoplist cgi-bin ?

cache deny all
# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/spool/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

# Hide Proxy from destination server
# Needed to share sessions
via off
forwarded_for off
request_header_access Allow allow all 
request_header_access Authorization allow all 
request_header_access WWW-Authenticate allow all 
request_header_access Proxy-Authorization allow all 
request_header_access Proxy-Authenticate allow all 
request_header_access Cache-Control allow all 
request_header_access Content-Encoding allow all 
request_header_access Content-Length allow all 
request_header_access Content-Type allow all 
request_header_access Date allow all 
request_header_access Expires allow all 
request_header_access Host allow all 
request_header_access If-Modified-Since allow all 
request_header_access Last-Modified allow all 
request_header_access Location allow all 
request_header_access Pragma allow all 
request_header_access Accept allow all 
request_header_access Accept-Charset allow all 
request_header_access Accept-Encoding allow all 
request_header_access Accept-Language allow all 
request_header_access Content-Language allow all 
request_header_access Mime-Version allow all 
request_header_access Retry-After allow all 
request_header_access Title allow all 
request_header_access Connection allow all 
request_header_access Proxy-Connection allow all 
request_header_access User-Agent allow all 
request_header_access Cookie allow all 
request_header_access All deny all

Is there a way to check if my connections are reaching the VPS?

I did not find a squid log beyond its access.log, which says nothing about my home connection;

Best Answer

Turns out it was an issue with the Linode datacenter.

traceroute pointed that the packets were being dropped uppon arrival on the Linode Atlanta Datacenter.

Note: Server Public IP and Hostname intentionally omitted

$ sudo traceroute -p 3128 HOSTNAME --tcp
traceroute to HOSTNAME (PUBLIC_IP), 30 hops max, 60 byte packets
1 192.168.0.254 (192.168.0.254) 0.922 ms 1.337 ms 2.016 ms
2 200.150.94.3 (200.150.94.3) 11.970 ms 12.337 ms 12.699 ms
3 trunk11-src1km3a-src1cos.copel.net (200.150.92.107) 11.980 ms 12.928 ms 12.962 ms
4 177.84.164.33 (177.84.164.33) 14.230 ms 15.943 ms 16.306 ms
5 149.3.181.42 (149.3.181.42) 17.045 ms 17.889 ms 18.127 ms
6 xe-7-2-1.ashburn2.ash.seabone.net (195.22.199.187) 171.647 ms xe-0-1-2.ashburn2.ash.seabone.net (89.221.40.3) 163.118 ms 161.458 ms
7 xe-7-0-1.ashburn2.ash.seabone.net (195.22.199.189) 163.412 ms xe-2-2-2.ashburn2.ash.seabone.net (195.22.199.183) 159.763 ms xe-7-0-1.ashburn2.ash.seabone.net (195.22.199.189) 162.821 ms
8 ae13.er2.iad10.us.zip.zayo.com (64.125.12.1) 161.005 ms 160.879 ms 161.481 ms
9 ae7.er1.iad10.us.zip.zayo.com (64.125.25.49) 146.142 ms 144.668 ms 146.361 ms
10 ae6.cr1.dca2.us.zip.zayo.com (64.125.20.117) 147.399 ms 147.831 ms 148.792 ms
11 ae2.mpr4.atl6.us.zip.zayo.com (64.125.25.46) 150.200 ms ae4.mpr3.atl6.us.zip.zayo.com (64.125.31.197) 144.751 ms ae2.mpr4.atl6.us.zip.zayo.com (64.125.25.46) 146.808 ms
12 ae4.mpr3.atl6.us.zip.zayo.com (64.125.31.197) 143.534 ms 143.451 ms 146.218 ms
13 * * *
...
30 * * *

As noted before, on port 53128 it went through.

$ sudo traceroute -p 53128 HOSTNAME --tcp
traceroute to HOSTNAME (PUBLIC_IP), 30 hops max, 60 byte packets
1 192.168.0.254 (192.168.0.254) 0.931 ms 1.458 ms 1.968 ms
2 200.150.94.3 (200.150.94.3) 12.079 ms 12.222 ms 12.362 ms
3 trunk11-src1km3a-src1cos.copel.net (200.150.92.107) 51.764 ms 52.274 ms 52.466 ms
4 177.84.164.33 (177.84.164.33) 14.533 ms 14.910 ms 15.449 ms
5 149.3.181.42 (149.3.181.42) 16.408 ms 17.265 ms 17.720 ms
6 xe-2-2-2.ashburn2.ash.seabone.net (195.22.199.183) 171.688 ms xe-7-2-1.ashburn2.ash.seabone.net (195.22.199.187) 161.815 ms xe-0-1-2.ashburn2.ash.seabone.net (89.221.40.3) 162.661 ms
7 xe-2-3-1.ashburn2.ash.seabone.net (195.22.199.181) 163.044 ms xe-2-2-2.ashburn2.ash.seabone.net (195.22.199.183) 160.476 ms xe-7-0-2.ashburn2.ash.seabone.net (195.22.199.185) 161.247 ms
8 ae13.er2.iad10.us.zip.zayo.com (64.125.12.1) 162.917 ms 162.569 ms 164.899 ms
9 ae7.er1.iad10.us.zip.zayo.com (64.125.25.49) 149.290 ms 147.938 ms ae6.cr1.dca2.us.zip.zayo.com (64.125.20.117) 150.054 ms
10 ae2.mpr4.atl6.us.zip.zayo.com (64.125.25.46) 150.416 ms ae6.cr1.dca2.us.zip.zayo.com (64.125.20.117) 145.685 ms 146.062 ms
11 ae4.mpr3.atl6.us.zip.zayo.com (64.125.31.197) 145.271 ms ae2.mpr4.atl6.us.zip.zayo.com (64.125.25.46) 146.286 ms ae4.mpr3.atl6.us.zip.zayo.com (64.125.31.197) 143.456 ms
12 128.177.104.134.IPYX-092136-ZYO.zip.zayo.com (128.177.104.134) 143.646 ms ae4.mpr3.atl6.us.zip.zayo.com (64.125.31.197) 145.377 ms 143.066 ms
13 128.177.104.134.IPYX-092136-ZYO.zip.zayo.com (128.177.104.134) 147.494 ms 148.962 ms 148.619 ms
14 router1-atl.linode.com (64.22.106.10) 148.807 ms 146.101 ms PRIVATE_HOSTNAME.members.linode.com (PUBLIC_IP) 145.706 ms

With that I was able to open a support ticket and got my definitely answer. I am glad it was not an issue with my server.

Hello there,

We've investigated further and it turns out this is actually an ongoing issue with our Atlanta data center, so we apologize we couldn't get this answer to you sooner. Currently we are working on getting this port un-blocked but we wouldn't be able to give a time estimate for that.

In the mean time, will that different port work for you? I could also migrate you to a different datacenter if you want with a working 3128 port -- perhaps Fremont, CA, or Newark, NJ?:

https://www.linode.com/speedtest

Let us know.

Regards, Roland Linode Support Team

Related Topic