Cisco IOS – Logging Source Interface

THE PROBLEM:
It turns out we ran into a node limit on the FWSM. Evidently you can be within your rule limit but hit the node ceiling. This doc https://supportforums.cisco.com/docs/DOC-8786 details the compilation memory exhaustion issue that hit us. Quoting:

This command show np 3 acl stats in the context in question will show if the total nodes is reached. This limit may be reached even before the ACL limit is reached. Each ACE may take a minimum of 2 nodes to a maximum up to to 5 nodes depending on where the ACL is being called. The ACL that is tied to MPF (modular policy framework) may take up more nodes than the ACL that is tied to a NAT or to the access-group. There is no way to calculate the number of nodes. The best way to monitor this is to regularly look at the above output to make sure the node count is not exceeded.

RESOLUTION:
1. Power down the secondary FWSM and power on the primary.
2. Revise our configuration by examining which rules could be removed (stale rules and recent changes allowing certain Development environments access to Production) in order to bring down the rule and therefore node count.
3. Ensure the new config compiles then reset the primary FWSM.
4. Verify the FWSM comes up clean with no errors.
5. Power down the primary FWSM and power up the secondary FWSM.
6. Repeat steps 3 and 4 with the secondary FWSM then power it down.
7. Power on the primary, ensure it's clean, power on the secondary.
8. Ensure the primary's config is copied to the secondary and that the primary is Active and sees the secondary as Standby.

MITIGATION:
1. We upgraded our development environment (2 x 6509 with Active/Standby FWSM) to FWSM version 4.1(13) from 3.2(23). The upgrade to the 4.x train increased the maximum node count for us from 28,356 to 38,439. More information: http://www.cisco.com/en/US/prod/collateral/modules/ps2706/product_bulletin_c25-478751.html.
2. We will upgrade our Production FWSM during the next change window.
3. We implemented a Kiwi CatTools job to run the show np 3 acl stats command in our contexts and e-mail us a daily report. Whenever we make a rule change we also run this command to ensure we're within node and rule limits.

It's interesting behavior that the FWSM would allow us to continue adding rules. It only failed when it was power cycled and tried to compile the rule set. Lessons learned!

Best practice wise - should I let the router or the ASA handle NAT (Overloading)?

In the most general of design best practices NAT is performed between an inside and outside network. NAT overloading is generally performed at the edge when there is limited public IP address space. You can learn more about NAT overloading, also known as Port Address Translation or PAT, in RFC 2663 (PAT is referred to as Network Address Port Translation (NAPT) in section 4.1.2).

In this particular scenario you can argue that you have two inside and outside networks and will need to perform some form of NAT on both the ASA (whether that is the NAT overloading you're using now, NAT exemption, static NAT, etc) and the Cisco Router.

I can ping the 172.16.2.2 interface but not 172.16.2.1 from a pc connected to one of the layer 2 switches (proves intervlan routing is working -- i have a 172.20.100.8 address on the PC). Why can't I ping 172.16.2.1 from a PC but I can from the Layer 3 Switch?

The ASA 172.16.2.2 is receiving the ICMP echo-request but does not have a route back to 172.20.100.0/27. The echo-reply is actually being forwarded to the Router 172.16.1.1 via the default route.

And most of all -- Why can't I get out to the Internet from the Layer 3 switch?

Currently your ASA and Cisco Router do not have routes to internal devices other than their connected routes.

Your ASA configuration:

route outside 0.0.0.0 0.0.0.0 172.16.1.1 1

This will provide a default route via the outside interface, but how will the ASA know how to reach subnets residing behind the Layer 3 Distribution Switch?

You'll need to add routes to the internal subnets via the inside interface using the Layer 3 Distribution Switch as the next-hop IP address.

ASA static routing example:

route inside 172.19.12.0 255.255.255.240 172.16.2.2
route inside 172.19.3.0 255.255.255.0 172.16.2.2
route inside 172.20.100.0 255.255.255.224 172.16.2.2

Further reading: ASA static routing

Your Cisco Router's configuration:

ip route 0.0.0.0 0.0.0.0 200.200.200.200

Additionally, how will your border router know how to reach subnets other than it's connected routes, and the catch all default route via the outside interface's next-hop address 200.200.200.200?

Router static routing example:

ip route 172.19.12.0 255.255.255.240 172.16.1.10
ip route 172.19.3.0 255.255.255.0 172.16.1.10
ip route 172.19.100.0 255.255.255.224 172.16.1.10
ip route 172.16.2.0 255.255.255.224 172.16.1.10

Further reading: ISR static routing

I cannot get an ip address right now from the DHCP server (Windows). Any insight into why?

Ensure you have end-to-end IP reachability between the client(s) sending DHCP discover messages and the DHCP server.

From what I can gather from your topology and configuration, the subnets 172.19.3.0/24, 172.19.12.0/28 and 172.20.100.0/27 should have no issues connecting to each other (assuming they are configured to use their respective default gateways) from a networking perspective.

You can remove the ip helper-address syntax from the SVI 100 given that the DHCP server is on the same segment and that command is used for a DHCP server(s) that is on a different segment.

interface Vlan100
ip address 172.20.100.1 255.255.255.224
ip helper-address 172.20.100.27