I am trying to replace our existing Virtual Chassis way of doing mutli-chassis LAG with EVPN multi-homing using VXLAN. My understanding is we should be able to replace any LAG we have with evpn-multihoming.
The reason is I want to get rid of the single point of failure of virtual-chass' shared route engine (we've had some issues) and this is part of a stepping stone to moving to a spine-leaf network, using VXLAN. So it makes sense to leverage this now.
I am trying to achieve what should be a fairly basic setup – two QFX 5110 switches with an ESI LAG, and EVPN to share MACs between the two independent switches.
The Problem
Fundamentally, my problem is I can't get remote MAC learning to work over EVPN.
- As soon as I add any VXLAN VNI config to my existing VLAN100, my
ethernet-switching tablegoes completely empty, with no MACs showing. And no local bridging of packets. Connectivity is completely lost.
{master:0}
root@core-01> show ethernet-switching table
{master:0}
root@core-01>
- If I remove the VXLAN config from VLAN100, MACs show in the
ethernet-switching tableand I get local bridging to work, but not remote.- I get some routes shared over BGP EVPN, but I only get type 1, 3 or 4 routes, never type 2.
- My understanding is I need type 2 routes to learn the actual MACs.
- My vtep interface shows as a member of vlan 100, but never listed as a forwarding option for any of the MACs.
Setup Details
- Using OSPF to learn loopbacks
- Using iBGP as my underlay.
- Only using a single EVPN instance.
- Not using any routing-instances/VRFs – just the default global table for now.
- Using Junos 17.3R1.10
Troubleshooting Output
Not sure exactly what is most relevant here, but I've included some basic info below
root@core-01> show route receive-protocol bgp 10.10.216.170
inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
3:10.10.216.170:100::100::10.10.216.170/248 IM
* 10.10.216.170 100 I
4:10.10.216.170:0::650000100113044501:10.10.216.170/296 ES
* 10.10.216.170 100 I
default-switch.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
3:10.10.216.170:100::100::10.10.216.170/248 IM
* 10.10.216.170 100 I
__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
4:10.10.216.170:0::650000100113044501:10.10.216.170/296 ES
* 10.10.216.170 100 I
root@core-01> show evpn instance
Intfs IRB intfs MH MAC addresses
Instance Total Up Total Up Nbrs ESIs Local Remote
__default_evpn__ 1
default-switch 114 2 0 0 1 1 0 0
{master:0}
{master:0}
root@core-01> show vlans
Routing instance VLAN name Tag Interfaces
default-switch VLAN100 100
ae1.0*
ge-0/0/2.0*
vtep.32769*
EDIT: A reboot has helped slightly
After rebooting both switches, I can now see my ethernet switching table populate and I am seeing EVPN type 2 routes, so MACs are being learned it seems. In the process of rebooting I've lost access to one of the two switches so I need to sort that out and continue troubleshooting.
Unfortunately even though things are looking better, I still don't have network connectivity.
{master:0}
root@core-02> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 5 entries, 5 learned
Routing instance : default-switch
Vlan MAC MAC Logical Active
name address flags interface source
VLAN100 00:15:fa:b6:76:41 D vtep.32769 10.10.216.169
VLAN100 00:15:fa:bc:18:01 D vtep.32769 10.10.216.169
VLAN100 00:1d:aa:56:4b:b8 D vtep.32769 10.10.216.169
VLAN100 c0:67:af:0a:9d:c0 D vtep.32769 10.10.216.169
VLAN100 e8:65:49:d5:43:40 D vtep.32769 10.10.216.169
And I'm seeing EVPN type 2 routes coming from the other switch:
root@core-02> show route receive-protocol bgp 10.10.216.169
inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
bgp.evpn.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
2:10.10.216.169:100::100::00:15:fa:b6:76:41/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::00:15:fa:bc:18:01/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::00:1d:aa:56:4b:b8/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::c0:42:d0:03:b1:01/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::c0:67:af:0a:9d:c0/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::e8:65:49:d5:43:40/304 MAC/IP
* 10.10.216.169 100 I
3:10.10.216.169:100::100::10.10.216.169/248 IM
* 10.10.216.169 100 I
default-switch.evpn.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
2:10.10.216.169:100::100::00:15:fa:b6:76:41/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::00:15:fa:bc:18:01/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::00:1d:aa:56:4b:b8/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::c0:42:d0:03:b1:01/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::c0:67:af:0a:9d:c0/304 MAC/IP
* 10.10.216.169 100 I
2:10.10.216.169:100::100::e8:65:49:d5:43:40/304 MAC/IP
* 10.10.216.169 100 I
3:10.10.216.169:100::100::10.10.216.169/248 IM
* 10.10.216.169 100 I
UPDATE 2: JunOS update fixed LACP Bug
-
As above, after rebooting both switches, I started seeing type 2 routes and started seeing MACs learned over the VTEP interface in my
ethernet-switching table. -
The next issue was no MAC learning on either LAG interface to the server. This was because despite
ge-0/0/1being up on both switches, traffic refused to pass and as a consequence LACP would not bring theae1interface up on either switch, or on the serverbond0. The LACP stats showedPort Disabledwhich is due to no begin event occurring within LACP: https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/show-lacp-interfaces.html -
I found that if I removed the LACP+LAG config (I didn't try just turning off LACP active mode) my configuring a separate VLAN 100 access port to use instead, I got connectivity to the server right away.
-
I know my LACP config on
ae1was working fine prior to enabling VXLAN+EVPN, so I put this down to another possible bug in JunOS, in the same way that I had to reboot the switches previously.
The Fix
Based on this I decided just to upgrade to the current JTAC recommended version.
This has worked. I upgraded:
- From: JunOS 17.3R1
- To: JunOS 20.2R2.11
Immediately after doing this on each switch, ae1 came up on that switch, and MAC learning on ae1 began working, and I have full reachability.
I can't confirm that a JunOS upgrade would have avoided the need to restart after enabling VXLAN VNI mapping on my VLAN100, but given I haven't read anything about having to restart, I'd say an upgrade would have fixed this too.
Here's my working EVPN routes:
{master:0}
root@core-02> show route protocol evpn
inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
bgp.evpn.0: 15 destinations, 15 routes (15 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1:10.10.216.170:0::650000100113044501::FFFF:FFFF/192 AD/ESI
*[EVPN/170] 02:38:08
Indirect
1:10.10.216.170:100::650000100113044501::0/192 AD/EVI
*[EVPN/170] 02:38:09
Indirect
3:10.10.216.170:100::100::10.10.216.170/248 IM
*[EVPN/170] 02:38:07
Indirect
4:10.10.216.170:0::650000100113044501:10.10.216.170/296 ES
*[EVPN/170] 02:38:09
Indirect
default-switch.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1:10.10.216.170:100::650000100113044501::0/192 AD/EVI
*[EVPN/170] 02:38:09
Indirect
3:10.10.216.170:100::100::10.10.216.170/248 IM
*[EVPN/170] 02:38:07
Indirect
__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1:10.10.216.170:0::650000100113044501::FFFF:FFFF/192 AD/ESI
*[EVPN/170] 02:38:08
Indirect
4:10.10.216.170:0::650000100113044501:10.10.216.170/296 ES
*[EVPN/170] 02:38:09
Indirect
{master:0}
root@core-02>
Configuration
The two main sources I've used to perform the configuration are:
- http://bgphelp.com/2017/04/01/vlan-over-evpn-vxlan-configuration/
- https://blog.noc.grnet.gr/2016/09/28/lab-on-evpn-vxlan-on-juniper-qfx5100-switches-3/
From core-01:
root@core-01# show vlans
VLAN100 {
vlan-id 100;
vxlan {
vni 100;
ingress-node-replication;
}
}
default {
vlan-id 1;
l3-interface irb.0;
}
root@core-01# show protocols
bgp {
group internal-peers {
type internal;
description "iBGP Peers";
local-address 10.10.216.169;
family evpn {
signaling;
}
authentication-key ## SECRET-DATA
peer-as 65000;
neighbor 10.10.216.170;
}
}
ospf {
reference-bandwidth 100g;
area 0.0.0.0 {
interface et-0/0/49.0 {
authentication {
md5 66 key ## SECRET-DATA
}
}
interface et-0/0/51.0 {
authentication {
md5 66 key ## SECRET-DATA
}
}
interface lo0.0 {
passive;
}
}
}
evpn {
vni-options {
vni 100 {
vrf-target export target:1:100; ## Warning: 'export' is deprecated
}
}
encapsulation vxlan;
multicast-mode ingress-replication;
extended-vni-list all;
}
policy-options {
policy-statement underlay-import {
term VXLAN100 {
from community VXLAN100;
then accept;
}
}
community VXLAN100 members target:1:100;
}
switch-options {
vtep-source-interface lo0.0;
route-distinguisher 10.10.216.169:100;
vrf-import underlay-import;
vrf-target target:65000:100;
}
root@core-01# show interfaces ge-0/0/1
description "Test - Server NIC#1";
ether-options {
802.3ad ae1;
}
root@core-01# show interfaces ae1
description "Test - Server";
mtu 9216;
esi {
00:65:00:00:10:01:13:04:45:01;
all-active;
}
aggregated-ether-options {
lacp {
active;
periodic fast;
system-id 65:00:00:10:01:00;
}
}
unit 0 {
family ethernet-switching {
interface-mode access;
vlan {
members VLAN100;
}
}
}
Best Answer
TLDR: Updated JunOS to the latest recommended version for QFX 5110 as per this JTAC document: https://kb.juniper.net/InfoCenter/index?page=content&id=KB21476&actp=METADATA#qfx_series
This is how I fixed the initial empty MAC table bug:
ethernet-switching table. Only had to reboot once for this bug to disappear.This is how I fixed the LAG/LACP bug:
I found that if I removed the LACP+LAG config (I didn't try just turning off LACP active mode) my configuring a separate VLAN 100 access port to use instead, I got connectivity to the server right away.
To fix this even with LACP enabled, I upgraded from: JunOS 17.3R1 to: JunOS 20.2R2.11
Immediately after doing this on each switch,
ae1came up on that switch, and MAC learning onae1began working, and I have full reachability.See original post for more details on working EVPN routes+MAC learning after applying this fix.