CentOS – Diagnose L2TP VPN Connection Failures

centosl2tpvpn

My local machine CentOS Linux release 7.7.1908 (Core) connects to my workplace VPN using LT2P IPSec with PSK. I have two gateways, but since both of them produce similar logs, I'm going to post one of them here.

When I try to connect to my VPN (gateway IP 103.7.249.66), the connection fails (which used to connect a few days ago, strangely enough) this is what I get in my /var/log/messages:

May 10 11:42:49 nid2_mig NetworkManager[1100]: <info>  [1589089369.6288] audit: op="connection-activate" uuid="20249836-0604-4082-b028-ec61462c2a8e" name="TigerIT1" pid=2653 uid=1002 result="success"
May 10 11:42:49 nid2_mig NetworkManager[1100]: <info>  [1589089369.6321] vpn-connection[0x563b09ece4f0,20249836-0604-4082-b028-ec61462c2a8e,"TigerIT1",0]: Started the VPN service, PID 6949
May 10 11:42:49 nid2_mig NetworkManager[1100]: <info>  [1589089369.6379] vpn-connection[0x563b09ece4f0,20249836-0604-4082-b028-ec61462c2a8e,"TigerIT1",0]: Saw the service appear; activating connection
May 10 11:42:49 nid2_mig NetworkManager[1100]: <info>  [1589089369.6811] vpn-connection[0x563b09ece4f0,20249836-0604-4082-b028-ec61462c2a8e,"TigerIT1",0]: VPN connection: (ConnectInteractive) reply received
May 10 11:42:49 nid2_mig journal: Check port 1701
May 10 11:42:49 nid2_mig NetworkManager: Redirecting to: systemctl restart ipsec.service
May 10 11:42:49 nid2_mig systemd: Stopping Internet Key Exchange (IKE) Protocol Daemon for IPsec...
May 10 11:42:49 nid2_mig whack: 002 shutting down
May 10 11:42:49 nid2_mig ipsec: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:49 nid2_mig libipsecconf[6977]: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:49 nid2_mig systemd: Stopped Internet Key Exchange (IKE) Protocol Daemon for IPsec.
May 10 11:42:49 nid2_mig systemd: Starting Internet Key Exchange (IKE) Protocol Daemon for IPsec...
May 10 11:42:49 nid2_mig addconn: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:49 nid2_mig libipsecconf[6983]: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:49 nid2_mig _stackmanager: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:49 nid2_mig libipsecconf[6989]: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:49 nid2_mig _stackmanager: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:49 nid2_mig libipsecconf[6994]: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:50 nid2_mig ipsec: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:50 nid2_mig libipsecconf[7254]: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:50 nid2_mig ipsec: nflog ipsec capture disabled
May 10 11:42:50 nid2_mig systemd: Started Internet Key Exchange (IKE) Protocol Daemon for IPsec.
May 10 11:42:50 nid2_mig libipsecconf[7299]: warning: could not open include filename: '/etc/ipsec.d/*.conf'
May 10 11:42:50 nid2_mig NetworkManager: 002 listening for IKE messages
May 10 11:42:50 nid2_mig NetworkManager: 002 forgetting secrets
May 10 11:42:50 nid2_mig NetworkManager: 002 loading secrets from "/etc/ipsec.secrets"
May 10 11:42:50 nid2_mig NetworkManager: 002 loading secrets from "/etc/ipsec.d/ipsec.nm-l2tp.secrets"
May 10 11:42:50 nid2_mig NetworkManager: debugging mode enabled
May 10 11:42:50 nid2_mig NetworkManager: end of file /var/run/nm-l2tp-20249836-0604-4082-b028-ec61462c2a8e/ipsec.conf
May 10 11:42:50 nid2_mig NetworkManager: Loading conn 20249836-0604-4082-b028-ec61462c2a8e
May 10 11:42:50 nid2_mig NetworkManager: starter: left is KH_DEFAULTROUTE
May 10 11:42:50 nid2_mig NetworkManager: conn: "20249836-0604-4082-b028-ec61462c2a8e" labeled_ipsec=0
May 10 11:42:50 nid2_mig NetworkManager: conn: "20249836-0604-4082-b028-ec61462c2a8e" modecfgdns=(null)
May 10 11:42:50 nid2_mig NetworkManager: conn: "20249836-0604-4082-b028-ec61462c2a8e" modecfgdomains=(null)
May 10 11:42:50 nid2_mig NetworkManager: conn: "20249836-0604-4082-b028-ec61462c2a8e" modecfgbanner=(null)
May 10 11:42:50 nid2_mig NetworkManager: conn: "20249836-0604-4082-b028-ec61462c2a8e" mark=(null)
May 10 11:42:50 nid2_mig NetworkManager: conn: "20249836-0604-4082-b028-ec61462c2a8e" mark-in=(null)
May 10 11:42:50 nid2_mig NetworkManager: conn: "20249836-0604-4082-b028-ec61462c2a8e" mark-out=(null)
May 10 11:42:50 nid2_mig NetworkManager: conn: "20249836-0604-4082-b028-ec61462c2a8e" vti_iface=(null)
May 10 11:42:50 nid2_mig NetworkManager: opening file: /var/run/nm-l2tp-20249836-0604-4082-b028-ec61462c2a8e/ipsec.conf
May 10 11:42:50 nid2_mig NetworkManager: loading named conns: 20249836-0604-4082-b028-ec61462c2a8e
May 10 11:42:50 nid2_mig NetworkManager: seeking_src = 1, seeking_gateway = 1, has_peer = 1
May 10 11:42:50 nid2_mig NetworkManager: seeking_src = 0, seeking_gateway = 1, has_dst = 1
May 10 11:42:50 nid2_mig NetworkManager: dst  via 192.168.68.1 dev wlp2s0 src  table 254
May 10 11:42:50 nid2_mig NetworkManager: set nexthop: 192.168.68.1
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.68.0 via  dev wlp2s0 src 192.168.68.108 table 254
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.122.0 via  dev virbr0 src 192.168.122.1 table 254
May 10 11:42:50 nid2_mig NetworkManager: dst 127.0.0.0 via  dev lo src 127.0.0.1 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 127.0.0.0 via  dev lo src 127.0.0.1 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 127.0.0.1 via  dev lo src 127.0.0.1 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 127.255.255.255 via  dev lo src 127.0.0.1 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.68.0 via  dev wlp2s0 src 192.168.68.108 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.68.108 via  dev wlp2s0 src 192.168.68.108 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.68.255 via  dev wlp2s0 src 192.168.68.108 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.122.0 via  dev virbr0 src 192.168.122.1 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.122.1 via  dev virbr0 src 192.168.122.1 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.122.255 via  dev virbr0 src 192.168.122.1 table 255 (ignored)
May 10 11:42:50 nid2_mig NetworkManager: seeking_src = 1, seeking_gateway = 0, has_peer = 1
May 10 11:42:50 nid2_mig NetworkManager: seeking_src = 1, seeking_gateway = 0, has_dst = 1
May 10 11:42:50 nid2_mig NetworkManager: dst 192.168.68.1 via  dev wlp2s0 src 192.168.68.108 table 254
May 10 11:42:50 nid2_mig NetworkManager: set addr: 192.168.68.108
May 10 11:42:50 nid2_mig NetworkManager: seeking_src = 0, seeking_gateway = 0, has_peer = 1
May 10 11:42:50 nid2_mig NetworkManager: 002 "20249836-0604-4082-b028-ec61462c2a8e" #1: initiating Main Mode
May 10 11:42:50 nid2_mig NetworkManager: 104 "20249836-0604-4082-b028-ec61462c2a8e" #1: STATE_MAIN_I1: initiate
May 10 11:42:50 nid2_mig NetworkManager: 106 "20249836-0604-4082-b028-ec61462c2a8e" #1: STATE_MAIN_I2: sent MI2, expecting MR2
May 10 11:42:50 nid2_mig NetworkManager: 108 "20249836-0604-4082-b028-ec61462c2a8e" #1: STATE_MAIN_I3: sent MI3, expecting MR3
May 10 11:42:50 nid2_mig NetworkManager: 002 "20249836-0604-4082-b028-ec61462c2a8e" #1: Peer ID is ID_IPV4_ADDR: '103.7.249.66'
May 10 11:42:50 nid2_mig NetworkManager: 004 "20249836-0604-4082-b028-ec61462c2a8e" #1: STATE_MAIN_I4: ISAKMP SA established {auth=PRESHARED_KEY cipher=3des_cbc_192 integ=sha group=MODP1024}
May 10 11:42:50 nid2_mig NetworkManager: 002 "20249836-0604-4082-b028-ec61462c2a8e" #2: initiating Quick Mode PSK+ENCRYPT+PFS+UP+IKEV1_ALLOW+SAREF_TRACK+IKE_FRAG_ALLOW+ESN_NO {using isakmp#1 msgid:a6c5fe68 proposal=AES_CBC_256-HMAC_SHA1_96, AES_CBC_128-HMAC_SHA1_96, 3DES_CBC-HMAC_SHA1_96 pfsgroup=MODP1024}
May 10 11:42:50 nid2_mig NetworkManager: 117 "20249836-0604-4082-b028-ec61462c2a8e" #2: STATE_QUICK_I1: initiate
May 10 11:42:50 nid2_mig NetworkManager: 010 "20249836-0604-4082-b028-ec61462c2a8e" #2: STATE_QUICK_I1: retransmission; will wait 0.5 seconds for response
May 10 11:42:51 nid2_mig NetworkManager: 010 "20249836-0604-4082-b028-ec61462c2a8e" #2: STATE_QUICK_I1: retransmission; will wait 1 seconds for response
May 10 11:42:52 nid2_mig NetworkManager: 010 "20249836-0604-4082-b028-ec61462c2a8e" #2: STATE_QUICK_I1: retransmission; will wait 2 seconds for response
May 10 11:42:54 nid2_mig NetworkManager: 010 "20249836-0604-4082-b028-ec61462c2a8e" #2: STATE_QUICK_I1: retransmission; will wait 4 seconds for response
May 10 11:42:58 nid2_mig NetworkManager: 010 "20249836-0604-4082-b028-ec61462c2a8e" #2: STATE_QUICK_I1: retransmission; will wait 8 seconds for response
May 10 11:43:00 nid2_mig journal: g_dbus_method_invocation_take_error: assertion 'error != NULL' failed
May 10 11:43:00 nid2_mig NetworkManager[1100]: <info>  [1589089380.2142] vpn-connection[0x563b09ece4f0,20249836-0604-4082-b028-ec61462c2a8e,"TigerIT1",0]: VPN plugin: state changed: stopped (6)
May 10 11:43:00 nid2_mig NetworkManager[1100]: <info>  [1589089380.2161] vpn-connection[0x563b09ece4f0,20249836-0604-4082-b028-ec61462c2a8e,"TigerIT1",0]: VPN service disappeared
May 10 11:43:00 nid2_mig NetworkManager[1100]: <warn>  [1589089380.2168] vpn-connection[0x563b09ece4f0,20249836-0604-4082-b028-ec61462c2a8e,"TigerIT1",0]: VPN connection: failed to connect: 'Message recipient disconnected from message bus without replying'

There was no .conf file under /etc/ipsec.d/ in my system.

Traceroute on the IP:

traceroute to 103.7.249.66 (103.7.249.66), 30 hops max, 60 byte packets
 1  gateway (192.168.68.1)  6.709 ms  6.734 ms  6.703 ms
 2  192.168.0.1 (192.168.0.1)  7.331 ms  7.401 ms  7.390 ms
 3  10.0.0.1 (10.0.0.1)  10.848 ms  10.834 ms  10.811 ms
 4  228.51.103-1-baninetworks.com (103.51.228.1)  10.786 ms  10.765 ms  10.739 ms
 5  220.152.112.213 (220.152.112.213)  8.062 ms  8.091 ms  10.269 ms
 6  103.7.248.109 (103.7.248.109)  15.651 ms  14.175 ms  14.188 ms
 7  * * *
 8  * * *
 9  * * *
10  * * *
.........

Local IP Routing Table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.68.1    0.0.0.0         UG    600    0        0 wlp2s0
192.168.68.0    0.0.0.0         255.255.255.0   U     600    0        0 wlp2s0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

I was using nm-connection-editor to create / update VPN connections. I haven't fiddled with anything under IPSec or PPP settings. Current configurations look like the following:

L2TP IPsec Settings
L2TP PPP Options

I have tried deleting and creating the connection several times. Sometimes re-configuring the connection works on Windows, but didn't help in CentOS case.

My machine is connected to internet via WiFi mesh router, however, my other machine (Windows) and my phones (Android) are also connected in the same network and I can connect to VPN from those devices. I haven't changed anything related to IP forwarding or MTU. Tried reaching my network admins, but they could not tell much, since the connection could not be established. However, I can ask them any specific question if I knew what to ask.

I do not think this is related to my ISP since I can connect from other operating systems.

I want to investigate what's going wrong. I have very little knowledge on network communication interfaces, but I would like to understand more before trying out random fixes from the internet. Let me know if I could provide any more information.

Best Answer

From the logs, IPsec phase 1 (main mode) is successful, but phase 2 (quick mode) failed.

Perhaps the VPN server doesn't use Perfect Forward Secrecy (PFS) for phase 2 (quick mode). So try the "Disable PFS" checkbox.