While it's not the most elegant solution, you can address this problem at the DNS level by using BIND split views, which allow you to present different DNS information to different clients. Since your VPN clients are neatly segregated, the filtering will be simple. Setting up your zone files so that you don't have to make multiple entries for each server takes a little art, but is not too difficult. See this example, or the BIND9 documentation.
You can control address selection with /etc/gai.conf. The configuration file is well documented, and already contains the defaults, so you can just begin tweaking.
The interesting defaults here are:
label ::1/128 0
label ::/0 1
label 2002::/16 2
label ::/96 3
label ::ffff:0:0/96 4
precedence ::1/128 50
precedence ::/0 40
precedence 2002::/16 30
precedence ::/96 20
precedence ::ffff:0:0/96 10
The last line gives the lowest preference to all IPv4 addresses.
If you want to give a higher preference to all IPv4, you could change it to:
precedence ::ffff:0:0/96 100
If you only wanted to give higher preference to specific IPv4 addresses or blocks, you can specify them as well. Remember that you have to use an IPv4-mapped IPv6 in hex.
So, to give preference to 203.0.113.0/24 over all IPv6, you would add:
label ::ffff:cb00:7100/120 5
precedence ::ffff:cb00:7100/120 100
Restart running applications to have them pick up changes you make.
On Debian derived systems, /etc/gai.conf is already present. On Red Hat derived systems, it is absent, but a sample file is located at /usr/share/doc/glibc-common-*/gai.conf; just copy it to /etc.
Best Answer
ssh's-6parameter forces the client to connect with IPv6:ssh -6 hostnameIt is possible to stop
sshdfrom listening on IPv6 but that would simply cause an error instead of your slow connection problem.The exact reason why your connection alternates randomly between the two may be hard (and pointless) to pinpoint, but is probably related to the results (possibly their order) received by the DNS servers and local DNS caching.