Vpn – Cisco IPSec Pass-through on ASA 5505 not working

I have been busting my brain for a few days not and I have so far not been able to figure out what the issue here is.

The Problem:

I am unable to establish a Client-to-ASA IPSec tunnel from behind another ASA.

The situation:

The network situation is like this:

              +---------------------------------------------------------------------+
              |                              VIA S2S VPN                            |
              |                                                                     |
+-------------v-----+                                                               |
|                   |                                                               |
|                   |                                                               |
|     VLAN 3        |                                                               |
|     "office"      |                                                               |
|     10.20.50.0/24 +---------------+                                               |
|                   |               |                                               | 
+-------------------+           +---v----------------+                              |
                                |                    |               +---------------------------+
                                |                    |               |    Internet  |            |
                                |     ASA 5505       +--------------->              |            |
+-------------------+           |                    |               |              |            |
|                   |           |                    <------------------------+     |            |
|    VLAN 5         +----------->                    |    S2S VPN    |        |     |            |
|    "visitor"      |           |                    |               +---------------------------+
|    192.168.0.0/24 |           |                    |                        |     |
|                   |           +--------------------+                        |     |
+-------------------+                                                 +-------v-----------+
                                                      +------------+  |             |     |
                                                      |            +-->    ASA 5512X|     |
                                                      |  VLAN 99   |  |             |     |
                                                      |  "management" |             |     |
                                                      |  10.20.99.0/24|             |     |
                                                      |            |  |             |     |
                                                      |            |  |             |     |
                                                      +------------+  +-------------------+
                                                                                    |
                                                                   +----------------v+
                                                                   |                 |
                                                                   |   VLAN 10       |
                                                                   |   "servers"     |
                                                                   |   10.20.30.0/24 |
                                                                   |                 |
                                                                   +-----------------+

I have the site to site VPN tunnel working and if you are in the "office" vlan you can access "servers" with no issues.

What I am not able to do is to establish a client to site IPsec tunnel either from Win, MacOS, or Linux while being in either "office" or "visitor". I know that the remote end, as well as my local configuration is OK because of:

• It worked until the previous ASA died (was not able to salvage the config)
• It works from home
• It works when tethered via mobile phone

What happens:

(I can provide a more detailed debug log if it helps)

➜  ~ sudo vpnc-connect --dpd-idle 0 --debug 1 --local-port 10000 ~/config.conf

vpnc version 0.5.3r550-3
IKE SA selected psk+xauth-3des-sha1
NAT status: this end behind NAT? YES -- remote end behind NAT? no
got address 10.xx.xx.xx
received notice of type  (ISAKMP_N_INVALID_ID_INFORMATION)(18), giving up


---!!!!!!!!! entering phase2_fatal !!!!!!!!!---


vpnc-connect: quick mode response rejected:  (ISAKMP_N_INVALID_MESSAGE_ID)(9)
this means the concentrator did not like what we had to offer.
Possible reasons are:
  * concentrator configured to require a firewall
     this locks out even Cisco clients on any platform except windows
     which is an obvious security improvement. There is no workaround (yet).
  * concentrator configured to require IP compression
     this is not yet supported by vpnc.
     Note: the Cisco Concentrator Documentation recommends against using
     compression, except on low-bandwith (read: ISDN) links, because it
     uses much CPU-resources on the concentrator

What you do know? / What have you tried?:

• It did not work in the most bare bones setup setup (Interface security, IP, default route, NAT)
• As far as I can tell it is not an issue with firewall dropping packets
  • I can see packets going both ways with WireShark
  • Packet counters on specific "pass" firewall rules are increasing both directions
  • All "trace-packet" commands seemed to give an OK result
• I added a "allow any any" rule as a test, no change in behavior
• I added a deny rule for the UDP ports -> Different error (expected)
• inspect ipsec-pass-thru on the default global policy does not seem to make any difference if it is present or not
• Site-to-Site VPN tunnel is NOT interfering as most of the testing was done before ANY configuration related to that was added
• I have read the config for some of our other sites and I cant find any statements that seem to explain it

I am presuming that the ASA is somehow modifying the packets in a way that breaks some checksum but I have so far been unable to convince it to no do that.