Best practice wise - should I let the router or the ASA handle NAT
(Overloading)?
In the most general of design best practices NAT is performed between an inside and outside network. NAT overloading is generally performed at the edge when there is limited public IP address space. You can learn more about NAT overloading, also known as Port Address Translation or PAT, in RFC 2663 (PAT is referred to as Network Address Port Translation (NAPT) in section 4.1.2).
In this particular scenario you can argue that you have two inside and outside networks and will need to perform some form of NAT on both the ASA (whether that is the NAT overloading you're using now, NAT exemption, static NAT, etc) and the Cisco Router.
I can ping the 172.16.2.2 interface but not 172.16.2.1 from a pc
connected to one of the layer 2 switches (proves intervlan routing is
working -- i have a 172.20.100.8 address on the PC). Why can't I ping
172.16.2.1 from a PC but I can from the Layer 3 Switch?
The ASA 172.16.2.2 is receiving the ICMP echo-request but does not have a route back to 172.20.100.0/27. The echo-reply is actually being forwarded to the Router 172.16.1.1 via the default route.
And most of all -- Why can't I get out to the Internet from the Layer 3 switch?
Currently your ASA and Cisco Router do not have routes to internal devices other than their connected routes.
Your ASA configuration:
route outside 0.0.0.0 0.0.0.0 172.16.1.1 1
This will provide a default route via the outside interface, but how will the ASA know how to reach subnets residing behind the Layer 3 Distribution Switch?
You'll need to add routes to the internal subnets via the inside interface using the Layer 3 Distribution Switch as the next-hop IP address.
ASA static routing example:
route inside 172.19.12.0 255.255.255.240 172.16.2.2
route inside 172.19.3.0 255.255.255.0 172.16.2.2
route inside 172.20.100.0 255.255.255.224 172.16.2.2
Further reading: ASA static routing
Your Cisco Router's configuration:
ip route 0.0.0.0 0.0.0.0 200.200.200.200
Additionally, how will your border router know how to reach subnets other than it's connected routes, and the catch all default route via the outside interface's next-hop address 200.200.200.200?
Router static routing example:
ip route 172.19.12.0 255.255.255.240 172.16.1.10
ip route 172.19.3.0 255.255.255.0 172.16.1.10
ip route 172.19.100.0 255.255.255.224 172.16.1.10
ip route 172.16.2.0 255.255.255.224 172.16.1.10
Further reading: ISR static routing
I cannot get an ip address right now from the DHCP server (Windows).
Any insight into why?
Ensure you have end-to-end IP reachability between the client(s) sending DHCP discover messages and the DHCP server.
From what I can gather from your topology and configuration, the subnets 172.19.3.0/24, 172.19.12.0/28 and 172.20.100.0/27 should have no issues connecting to each other (assuming they are configured to use their respective default gateways) from a networking perspective.
You can remove the ip helper-address syntax from the SVI 100 given that the DHCP server is on the same segment and that command is used for a DHCP server(s) that is on a different segment.
interface Vlan100
ip address 172.20.100.1 255.255.255.224
ip helper-address 172.20.100.27
If you can be sure of your cabling then I would recommend implementing spanning-tree bpdufilter on the interface towards the firewall since you have already configured the interface as an edge port. This does not provide any loop protection, but the ASA (non-5505 as indicated) is incapable of creating a L2 loop and your stated goal is to remove BPDU's from the interface.
Configuring root-guard on the interface will not be useful since the ASA can never send any BPDU, and especially not a superior BPDU. If you do not choose to configure BPDU filtering, then BPDU-Guard should be configured either by default (spanning-tree portfast bpduguard default) or specifically on the interface. This will prevent any other spanning-tree capable device from being connected to that interface.
Further, if you do not have an untagged (native) interface on the firewall, allowing the native vlan on the trunk is not required and your allowed vlan list could be shortened to exclude 99. Also you could set the interface to not negotiate the access/trunking mode with switchport nonegotiate and remove the configured access vlan.
Best Answer
Edited to reflect new information in the question:
Based on your multiple edits which finally give the whole picture (I hope), you can try something like this, but using the switch stacking capability in the alternative configuration will give you a better solution.
Your connection to the ISP will need to be on the same subnet as the ISP.
SW1:
SW2:
This allows the ISP HSRP to work since the ISP routers need to be able to talk to each other on the same VLAN so that the ISP HSRP works.
Alternate (stacked) Configuration:
SW1/2 stack:
This configuration gives you all the benefits of the two separate switches but solves your problem of having two separate switches with separate IP addresses. The stack is configured as a single switch and is managed as a single device. Besides the data sharing, the 3850s can also do power sharing so that if one of them loses power, the other can keep both switches running. You will also save precious public IP addresses.