Cisco – Provide L3 gateways in a multi-tenant VXLAN environment with ASR9K and N9K

I'm trying to decode what you're doing, but I think I'm reading something like the following-

• You're using VXLAN-EVPN on the N9K's and the ASR9K's. The Nexus boxes are acting as standard VTEP's providing L2/L3 services while the ASR's are basically border leaves.
• The 3750's are pure L2 devices that pick up trunks from the upstream 9K's and break out access ports to downstream customers.

So - if we assume the above to be true then there are actually a few ways you can approach the problem. The q-in-vni approach mentioned isn't really one of them, though, as that's really more intended to tie together pure L2 environments. This would be applicable if, for example, if you had a client with a few 3750's spread across the network and you wanted to allow them to run a full set of VLAN's among them without having to provision each one on each VTEP. Put another way, there's no gateway involved.

Anyhow - back to what would work. Let's assume you assign a unique VXLAN VNI to each customer network you'll be provisioning. This VNI can show up on any of the VTEP's as needed. Wherever it appears you'll associate it to an SVI which, in turn, would be configured with the customer's vrf and an identical anycast address (i.e. the same default gateway will be present everywhere this customer's network appears). Keep in mind that wherever a given customer's vrf appears you'll also need to create an SVI to associate with an L3VNI. The SVI's don't need to be identically numbered across your VTEP's, as you'll be establishing a mapping on each box between VLAN and VNI numbers.

You'll also need to establish an underlay IP network amongst the N9K's and the ASR9K's. It's possible to do this with pure eBGP but I'd strongly recommend using a limited IGP deployment and then setting up iBGP - either with a full mesh or, preferably, a few route reflectors. There's a great deal written about this on various design guides. There are also some considerations that come in if you'd like to use VPC between pairs of VTEP's to provide redundancy to your downstream clients.

So - the two ways in which you can now manage VLAN numbering on the VTEP is either using the traditional mechanism, which is to globally assign a given VLAN on VTEP-wide basis. You'd say, for example, that VLAN 100 gets mapped to VNI 10100 and then simply allow that VLAN over a trunk to a 3750 that contains said customer. The next customer's VLAN might be VLAN 101 that gets mapped to VNI 10101. Put another way, any reference to VLAN 100 means the same thing on any port on that box.

The other way this can be managed is with port-local VLAN mapping. In this case you can actually map the VLAN identifier on an individual port to any other VLAN on the box. In this case you might create a policy that says that every downstream customer is going to see VLAN 100 but that you'll actually map 100 on the customer facing port to whatever value you want on the box. As an example - perhaps you assign VLAN's 2000-3900 to individual customer VLAN's (..each mapped to their respective VNI, obviously). On the first customer-facing port you might map incoming VLAN 100 to local VLAN 2000. On the next customer-facing port the mapping might be VLAN 100 to local VLAN 2001, and so forth. This potentially simplifies client configuration (...everyone just uses VLAN100 from their point of view, after all) and hides the details of your network configuration.

Now - regardless of your choices above, the ASR side is actually not that big of a deal. The clients are actually routing to default gateways on the Nexus 9K's which, in turn, are using BGP-learned EVPN prefixes to send traffic on to the ASR - which can then be configured, for example, to cross-import these routes to another vrf with Internet access (...again, just an example). The opposite is also true (the N9K's are advertising routes and MAC addresses via BGP to both each other and the ASR's).

I've glossed over a number of details about EVPN design and implementation and would strongly recommend that you read some of the design and configuration guides out there to get a better feel for how these things work together. It's pretty approachable and familiar if you've done a fair amount with non-trivial BGP and self-managed MPLS but might be confusing if these things aren't in your background. It's also worth mentioning that the above solution is going to provide full L2 reachability within common VNI's for a customer and will provide routing between VNI's if a customer has multiple networks within their vrf.