Linux – Trouble getting transparent Squid forwarding proxy working with ssl

automated-testinglinuxlinux-networkingsquid

I have an environment consisting of four servers networked together. One server acts as the server, and the other three act as clients for running automated tests and Linux benchmarking using Phoromatic.

The four systems are all behind a corporate firewall. If I set the "http_proxy" and "https_proxy" environment variables on the clients, they can connect to the outside world and download tests and such, however they will not connect to the server as they try to connect to the local server using the proxy. Since I wanted to cache the package downloads, tests, etc… I set up a Squid proxy on the server system, and configured it as a transparent proxy, but it only works with http requests.

What I'd like to do is have the http requests handled via the cache, and forwarded to the parent proxy as needed. Obviously I can't decrypt the ssl sessions, but I can't figure out how to have the Squid proxy forward https requests to the parent proxy. Additionally, the squid proxy is running on the same box as the Phoromatic server, which is Web based but uses a user-configurable nonstandard port, but Squid likes to block requests to said port, even when it's added to the configuration as being allowed.

I would be OK with just having the clients use the corporate firewall directly for https and ftp requests and either just using the Squid cache for http requests, or ditch the Squid proxy altogether and have the clients set to not use the proxy for local hosts.

It's really frustrating me, since most of the time I'm great at hunting down information and making things work on my own, without having to pick anyone else's brain about it, but I guess I have a rather unique situation! And yes, I have tried the Phoronix forum for Phoromatic to no avail.

Servers are SuperMicro X8DTT dual chassis systems running Fedora 24. Network configuration consists of a GbE connection to a switch (used as the connection to the outside world) as well as two 10Gb on each system, also connected through a switch, but the 10Gb system is not connected to the outside world – they're used for bandwidth testing (The drivers for the 10Gb cards is what the system is set up to test)

Best Answer

I'll be short (yeah, it doesn't look short like at all, but otherwise it would be way longer and totally unreadable).

  • it doesn't look like you need proxy. like totally.
  • in a modern environment a cache ratio can be between zero and 40% (and I'm judging based on my proxies byte ratios), so if you want to save that amount of data, you can of course, use proxy. But consider this: in today's enterprise environment the role of a proxy is more of authorizing users on their way to the WAN access, than caching data. And that's the main reason for doubting your choice.
  • if you still need the proxy, it doesn't mean you have to decrypt the HTTPS. just let it live. it won't be cached, so what. It's in its design.
  • if you're still insisting on decrypting HTTPS - you can use sslBump technique. But this may be illegal in some countries, and furthermore this complicates things a lot. Like A LOT. I advise you not to go this way only for caching purposes.
  • don't serve local traffic via proxy: it adds latency, it loads the proxy, with unnecessary traffic (since it's cheap and LAN channels are way wider than WAN), it complicates the debugging and it adds parasite network dependencies, so it's unwise.
  • since I doubt you need proxy, I doubt even more you need a parent proxy. Looks like you're just having this thing .... you know, being into proxies. use one if you need it.
  • may be instead of the proxy you just need a fast and decent web-server, line nginx. So in a situation when your web-servers get overloaded it can act as a balancer for a farm with a l2-cache.
  • squid isn't scaling that well. for 10-gigs bandwidth you'll have to use SMP squid features, and this have it's downsides. Like unbalanced load on squid workers, SMP issues in squid internals, and so on. It may be solvable if you have previous experience with squid, but unlikely if you have set it up like for the first time.
  • finally, if you're deciding to stick with squid, it doesn't have to be transparent: you can configure WPAD for clients, and let servers decide how they should acces Internet.
Related Topic