Cisco – What path does a packet follow through a Cisco Nexus 5k/2k infrastructure

ciscocisco-nexus-2000cisco-nexus-5kpacket-path

I was told by someone that a packet going between two hosts connected to the same Nexus 2k would end up being passed all the way up to the 5k that is managing said 2k for logic to be applied and then passed back down to the 2k and out the port the destination host is connected to. That seems extremely inefficient to me, so I'm looking for someone to either confirm or deny the information I've been given.

Best Answer

The Nexus 2k isn't an actual switch, but a "fabric extender" - it functions as a separate, self-contained linecard for the Nexus 5k. The following CiscoLive PDF has great details on the Nexus 2k/5k architecture

It's clearly pointed out in page 56 that the 2k does not perform any local switching.

The Nexus 5k is what performs all of the forwarding, integrating the Nexus 2k fabric extenders remotely into its backplane via IEEE 802.1BR.

As a future point of advice, googling for "a day in the life of a packet on [x]" where [x] is your favorite piece of gear will often produce good documentation on the architecture of that gear. YMMV with this however, as not all vendors make this information public.

Related Solutions

Cisco – the benefit of a Cisco “top of rack” design

Keep in mind that a FEX inherently is a method by which to extend the fabric, thus the name. The ability to manage centrally while still having "line cards" distributed throughout the DC is the real reason to use a FEX. Drastically reducing cabling is valuable to anyone, technical or not, and the argument of being able to manage the entire infrastructure at fewer points is time savings, pure and simple.

One of your big doubts is that you're worried about single points of failure. All devices have the ability to be dual-homed, and in certain configurations, you can even establish virtual port channels with the Nexus 2K FEXs themselves.

Take a look at Cisco's documentation. You'll find that you can design a topology that is just as redundant as the "direct-cable" option you're considering, with less fuss.

Cisco – NX-OS: Importing already mirrored config into a switch-profile

The bug-id you are hitting is probably CSCuf56076, which is fixed in 6.0(2)N2(1). Unfortunately this is the best solution for this problem. Luckily it doesn't impact the traffic on the switches.

For debugging you can see which part of the config is in the global database:

show system internal csm info global-db seq-tbl

and which part is in the synced database:

show system internal csm info switch-profile <switch-profile-name> cfgd-db seq-tbl

When the sync-peer are removed you should make sure that both switches have the same config, and that the same parts of the config are in the switch-profile. Then re-add the sync-peer.

Be aware that, depending on what exactly goes wrong, this bug might re-appear after every reload of the switch. When we hit this bug we first performed the fix in the middle of the night to make sure there was no downtime. After getting more confident that traffic was not impacted we dared to fix these problems at slightly more convenient hours of the day (but still early in the morning) until Cisco released a version that fixed this.

Best Answer

Related Solutions

Cisco – the benefit of a Cisco “top of rack” design

Cisco – NX-OS: Importing already mirrored config into a switch-profile

Related Topic