Cisco – Backing up configs from a multi context firewall to Ciscoworks LMS Prime

THE PROBLEM:
It turns out we ran into a node limit on the FWSM. Evidently you can be within your rule limit but hit the node ceiling. This doc https://supportforums.cisco.com/docs/DOC-8786 details the compilation memory exhaustion issue that hit us. Quoting:

This command show np 3 acl stats in the context in question will show if the total nodes is reached. This limit may be reached even before the ACL limit is reached. Each ACE may take a minimum of 2 nodes to a maximum up to to 5 nodes depending on where the ACL is being called. The ACL that is tied to MPF (modular policy framework) may take up more nodes than the ACL that is tied to a NAT or to the access-group. There is no way to calculate the number of nodes. The best way to monitor this is to regularly look at the above output to make sure the node count is not exceeded.

RESOLUTION:
1. Power down the secondary FWSM and power on the primary.
2. Revise our configuration by examining which rules could be removed (stale rules and recent changes allowing certain Development environments access to Production) in order to bring down the rule and therefore node count.
3. Ensure the new config compiles then reset the primary FWSM.
4. Verify the FWSM comes up clean with no errors.
5. Power down the primary FWSM and power up the secondary FWSM.
6. Repeat steps 3 and 4 with the secondary FWSM then power it down.
7. Power on the primary, ensure it's clean, power on the secondary.
8. Ensure the primary's config is copied to the secondary and that the primary is Active and sees the secondary as Standby.

MITIGATION:
1. We upgraded our development environment (2 x 6509 with Active/Standby FWSM) to FWSM version 4.1(13) from 3.2(23). The upgrade to the 4.x train increased the maximum node count for us from 28,356 to 38,439. More information: http://www.cisco.com/en/US/prod/collateral/modules/ps2706/product_bulletin_c25-478751.html.
2. We will upgrade our Production FWSM during the next change window.
3. We implemented a Kiwi CatTools job to run the show np 3 acl stats command in our contexts and e-mail us a daily report. Whenever we make a rule change we also run this command to ensure we're within node and rule limits.

It's interesting behavior that the FWSM would allow us to continue adding rules. It only failed when it was power cycled and tried to compile the rule set. Lessons learned!

The ASA does not have a mechanism to mark peers as up or down. When DPD detects a peer is no longer available, any SA with that peer is torn down. When "interesting traffic" requires a new SA, the ASA goes through its normal phase 1 process, which means starting with the first peer in your crypto map and if a connection cannot be established, trying the second.

The 30 seconds of delay is a timing issue while the ASA "sorts things out" so to speak. You have traffic triggering a new SA, while DPD is tearing down the existing SA to the same peer, which creates a bit of a race condition that eventually sorts out.

If faster failover is desired, it usually entails maintaining both tunnels simultaneously and using GRE and dynamic routing to move traffic when a tunnel fails. However, that would require using routers instead of ASAs, since ASAs do not support GRE, and have some shortcomings with dynamic routing.