Best practice wise - should I let the router or the ASA handle NAT
(Overloading)?
In the most general of design best practices NAT is performed between an inside and outside network. NAT overloading is generally performed at the edge when there is limited public IP address space. You can learn more about NAT overloading, also known as Port Address Translation or PAT, in RFC 2663 (PAT is referred to as Network Address Port Translation (NAPT) in section 4.1.2).
In this particular scenario you can argue that you have two inside and outside networks and will need to perform some form of NAT on both the ASA (whether that is the NAT overloading you're using now, NAT exemption, static NAT, etc) and the Cisco Router.
I can ping the 172.16.2.2 interface but not 172.16.2.1 from a pc
connected to one of the layer 2 switches (proves intervlan routing is
working -- i have a 172.20.100.8 address on the PC). Why can't I ping
172.16.2.1 from a PC but I can from the Layer 3 Switch?
The ASA 172.16.2.2 is receiving the ICMP echo-request but does not have a route back to 172.20.100.0/27. The echo-reply is actually being forwarded to the Router 172.16.1.1 via the default route.
And most of all -- Why can't I get out to the Internet from the Layer 3 switch?
Currently your ASA and Cisco Router do not have routes to internal devices other than their connected routes.
Your ASA configuration:
route outside 0.0.0.0 0.0.0.0 172.16.1.1 1
This will provide a default route via the outside interface, but how will the ASA know how to reach subnets residing behind the Layer 3 Distribution Switch?
You'll need to add routes to the internal subnets via the inside interface using the Layer 3 Distribution Switch as the next-hop IP address.
ASA static routing example:
route inside 172.19.12.0 255.255.255.240 172.16.2.2
route inside 172.19.3.0 255.255.255.0 172.16.2.2
route inside 172.20.100.0 255.255.255.224 172.16.2.2
Further reading: ASA static routing
Your Cisco Router's configuration:
ip route 0.0.0.0 0.0.0.0 200.200.200.200
Additionally, how will your border router know how to reach subnets other than it's connected routes, and the catch all default route via the outside interface's next-hop address 200.200.200.200?
Router static routing example:
ip route 172.19.12.0 255.255.255.240 172.16.1.10
ip route 172.19.3.0 255.255.255.0 172.16.1.10
ip route 172.19.100.0 255.255.255.224 172.16.1.10
ip route 172.16.2.0 255.255.255.224 172.16.1.10
Further reading: ISR static routing
I cannot get an ip address right now from the DHCP server (Windows).
Any insight into why?
Ensure you have end-to-end IP reachability between the client(s) sending DHCP discover messages and the DHCP server.
From what I can gather from your topology and configuration, the subnets 172.19.3.0/24, 172.19.12.0/28 and 172.20.100.0/27 should have no issues connecting to each other (assuming they are configured to use their respective default gateways) from a networking perspective.
You can remove the ip helper-address syntax from the SVI 100 given that the DHCP server is on the same segment and that command is used for a DHCP server(s) that is on a different segment.
interface Vlan100
ip address 172.20.100.1 255.255.255.224
ip helper-address 172.20.100.27
Answering my own question to help future googlers. I spent about 3 hours on the phone with TAC; we finally got to the root cause of the issue.
The solution is to add a special NAT entry, which matches the IP address in the DNS A-Record when it arrives on the INSIDE interface.
object network DNS_NAT_masd1
description xlate A-Record DMZ src 1.195.18.182 to INSIDE src 10.195.18.182
host 1.195.18.182
nat (DMZ,INSIDE) static 10.195.18.182
When I asked for a pointer to documentation that describes why DNS translation works this way, the TAC lead said that he didn't know of any that described this behavior. The TAC lead also mentioned that with more code, the ASA would know to automatically translate the DNS A-Record without explicitly adding object network DNS_NAT_masd1; however, that is not how the dns keyword for ASA NAT works today. For reasons that still are not completely clear yet, the ASA requires the DNS A-Record IP to match the <proxy_addr> in the NAT statement, using syntax similar to this...
object network obj-EXAMPLE
description NAT object explicitly for translating DNS A-Records
host <proxy_addr>
nat (<REAL_INTF>,<PROXY_INTF>) static <real_addr> dns
The difficulty is that this configuration is exactly backwards for what you need to do if you're going to nat regular "data plane" IP traffic through the firewall.
This is the whole configuration that works...
object network DMZ_NAT_masd1
host 10.195.18.182
description xlate masd1 NAT DMZ src 10.195.18.182 to INSIDE src 192.168.11.101
object network INSIDE_NAT_masd1
host 10.195.18.182
description xlate masd1 NAT INSIDE src 10.195.18.182 to DMZ src 1.195.18.182
!!! DNS_NAT_masd1 is new
object network DNS_NAT_masd1
host 1.195.18.182
description xlate A-Record DMZ src 1.195.18.182 to INSIDE src 10.195.18.182
!
object network DMZ_NAT_masd1
nat (DMZ,INSIDE) static 192.168.11.101
object network INSIDE_NAT_masd1
nat (INSIDE,DMZ) static 1.195.18.182
!!! DNS_NAT_masd1 is new
object network DNS_NAT_masd1
nat (DMZ,INSIDE) static 10.195.18.182 dns
Best Answer
After some research I discovered the ASA does not allow pinging from one interface to another. So this explains why I was able to ping the inside interface when it was setup on the physical port but not when it was setup as the VLAN.
Edit: I'll elaborate. If my inside interface:
(i.e. GigabitEthernet0/2) is not setup as a VLAN and has an IP of (example) 1.1.1.2, I can ping this interface directly as long as I'm sourcing from the inside (and, of course, on the subnet).
However, if I were to setup a subinterface as the inside:
I would no longer be able to ping this address because it is treated as cross-interface pinging.
I found this and confirmed with a Cisco rep.
Apologies for the vague answer to begin with.