Cisco adaptive security appliance is dropping packets where SYN flag is not set

ajpcisco-asadmzmod-proxymod-proxy-ajp

We have an apache instance sitting inside our DMZ which is configured to proxy requests to an internal NATed tomcat instance inside our network. It works fine, but then all of a sudden requests from apache to the tomcat instance stop getting through with the following in the apache logs:

[error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

Investigating into the Cisco log viewer reveals the following:

Error Message %ASA-6-106015: Deny TCP (no connection) from IP_address/port to
IP_address/port flags tcp_flags on interface interface_name.
Explanation The adaptive security appliance discarded a TCP packet that has no associated connection in the adaptive security appliance connection table. The adaptive security appliance looks for a SYN flag in the packet, which indicates a request to establish a new connection. If the SYN flag is not set, and there is not an existing connection, the adaptive security appliance discards the packet.

Recommended Action None required unless the adaptive security appliance receives a large volume of these invalid TCP packets. If this is the case, trace the packets to the source and determine the reason these packets were sent.

All are machines are virtualised using VMware, and by default machines have been using the Intel E1000 emulated NIC. Our network administrator has changed this to a VMXNET3 driver in an attempt to correct the problem, we just have to wait and see if the problem persists as it's an intermittent problem.

Is there something else that could be causing this problem? This isn't the first service where we have had similar issues.

Our apache host is running Ubuntu 11.10 with a kernel version of 3.0.0-17-server. We have also had this issue on RHEL5 (5.8) running kernel 2.6.18-308.16.1.el5, this machine also has the E1000 NIC.

NOTE: I am not a network administrator and am a software architect and analyst programmer responsible for these systems.

Best Answer

Problem has been found to be the ASA closing persistent connections after a period of time, when it closes the connections it's also been configured to not send RST messages when a call is once again made.

To understand why this causes a problem I can illustrate it here.

  1. Apache creates first connection which succeeds.
  2. After a delay longer than ASA reset time the ASA closes the connection.
  3. Request is made and Apache tries sending on what it thinks is an open connection and times out after TimeOut - default 300 seconds
  4. Apache sends error to client

The problem here is amplified if there are a number of pooled connections still open. For example if Apache started with 5 pooled connections, and after the above closed one, it will still exhibit this behavior another 4 times before a client gets a successful request.

There are several ways to overcome this.

  1. Allow the ASA to send RST messages to clients it trusts.
  2. Set the configuration for mod_proxy:ProxyPass - keepalive to On
  3. Set the configuration for mod_proxy:ProxyPass - ttl to something lower than the firewalls reset time.

Don't try configuring mod_proxy:ProxyPass - timeout and mod_proxy:ProxyPass - connectiontimeout too low, as if you have any long running operations existing in your tomcat instance say for example any web services or ReST endpoints then they may start to fail if they take longer than this time.

Our solution is to do both of the first two options.

Related Topic