Cisco Switch Failover – Diverting Traffic to Standby ASA

THE PROBLEM:
It turns out we ran into a node limit on the FWSM. Evidently you can be within your rule limit but hit the node ceiling. This doc https://supportforums.cisco.com/docs/DOC-8786 details the compilation memory exhaustion issue that hit us. Quoting:

This command show np 3 acl stats in the context in question will show if the total nodes is reached. This limit may be reached even before the ACL limit is reached. Each ACE may take a minimum of 2 nodes to a maximum up to to 5 nodes depending on where the ACL is being called. The ACL that is tied to MPF (modular policy framework) may take up more nodes than the ACL that is tied to a NAT or to the access-group. There is no way to calculate the number of nodes. The best way to monitor this is to regularly look at the above output to make sure the node count is not exceeded.

RESOLUTION:
1. Power down the secondary FWSM and power on the primary.
2. Revise our configuration by examining which rules could be removed (stale rules and recent changes allowing certain Development environments access to Production) in order to bring down the rule and therefore node count.
3. Ensure the new config compiles then reset the primary FWSM.
4. Verify the FWSM comes up clean with no errors.
5. Power down the primary FWSM and power up the secondary FWSM.
6. Repeat steps 3 and 4 with the secondary FWSM then power it down.
7. Power on the primary, ensure it's clean, power on the secondary.
8. Ensure the primary's config is copied to the secondary and that the primary is Active and sees the secondary as Standby.

MITIGATION:
1. We upgraded our development environment (2 x 6509 with Active/Standby FWSM) to FWSM version 4.1(13) from 3.2(23). The upgrade to the 4.x train increased the maximum node count for us from 28,356 to 38,439. More information: http://www.cisco.com/en/US/prod/collateral/modules/ps2706/product_bulletin_c25-478751.html.
2. We will upgrade our Production FWSM during the next change window.
3. We implemented a Kiwi CatTools job to run the show np 3 acl stats command in our contexts and e-mail us a daily report. Whenever we make a rule change we also run this command to ensure we're within node and rule limits.

It's interesting behavior that the FWSM would allow us to continue adding rules. It only failed when it was power cycled and tried to compile the rule set. Lessons learned!

A reload is only required when changing encryption features or downgrading a permanent license. You shouldn't have to break your failover pair either. For a failover pair running >8.3(1), you only need a context license on the primary, and the secondary will inherit. Context licenses applied to both units in a failover pair will combine.