Cisco Total output drops on interface

ciscoerrorsswitch

we have c3850 trunk with other switch 4948 and its LACP 4x1G interface bonding (etherchannel) out of 4 1G we are getting Total output drops on 2 interface.

This is all started yesterday when i upgrade software on this switch before that everything was good. doesn't make sense. I have done this upgrade on other switches to but no issue so far. 

C3850#sh int g2/0/1
GigabitEthernet2/0/1 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 1c1d.86a9.3501 (bia 1c1d.86a9.3501)
  Description: coreSW_1_W_1of4_p
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 36/255, rxload 25/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:11, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 133157273
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 100659000 bits/sec, 61038 packets/sec
  5 minute output rate 144390000 bits/sec, 76852 packets/sec
     3445318149 packets input, 715126259616 bytes, 0 no buffer
     Received 827824 broadcasts (666658 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 666658 multicast, 0 pause input
     0 input packets with dribble condition detected
     4521665906 packets output, 1080744400610 bytes, 0 underruns
     133157273 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Counter:

C3850#sh int g2/0/1 counters

Port            InOctets    InUcastPkts    InMcastPkts    InBcastPkts
Gi2/0/1     718020111847     3459023952         668522         161527

Port           OutOctets   OutUcastPkts   OutMcastPkts   OutBcastPkts
Gi2/0/1    1084965964251     4539489816         109940          86365

Stats:

C3850#sh int g2/0/1 stats
GigabitEthernet2/0/1
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor      10458    2249522          0          0
             Route cache          0          0      18985    2870366
                   Total      10458    2249522      18985    2870366

EDIT:

Our total utilization on link is 400Mbps, and i am seeing all link using around ~100 Mbps individually (few bytes here and there).

I am not seeing any performance issue on network. this trunk is connected to other switch and they are also running same speed interface 4x1G. I have upgrade multiple switch with same software and they all working fine without any interface error.

Best Answer

Have you noticed a degradation in performance since doing the upgrade? If not, the issue could be that you have been experiencing the issue all along but your previous IOS version wasn't reporting the errors.

let's say you are experiencing a degradation in performance though. As you're seeing output drops this typically means that the links are experiencing congestion.

As you might already know, traffic is not equally split across all links in a port channel. For example, port channeling 4 x 1Gb links does not guarantee you 4Gb of throughput. The traffic is split across the links depending on what hashing algorithm you're using. Given this, it actually possible for one link to be maxed out while the other links remain dormant.

While we're on the topic of load balancing algorithms, the issue your experiencing could be related to CEF polarization.

Cisco IOS introduced a concept called unique-ID/universal-ID which helps avoid CEF polarization. This algorithm, called the universal algorithm (the default in current Cisco IOS versions), adds a 32-bit router-specific value to the hash function (called the universal ID - this is a randomly generated value at the time of the switch boot up that can can be manually controlled). This seeds the hash function on each router with a unique ID, which ensures that the same source/destination pair hash into a different value on different routers along the path. This process provides a better network-wide load-sharing and circumvents the polarization issue. This unique -ID concept does not work for an even number of equal-cost paths due to a hardware limitation, but it works perfectly for an odd number of equal-cost paths. In order to overcome this problem, Cisco IOS adds one link to the hardware adjacency table when there is an even number of equal-cost paths in order to make the system believe that there is an odd number of equal-cost links. In order to configure a customized value for the universal ID, use: 

6500(config)ip cef load-sharing algorithm universal <id>

Another possible cause is micro bursting which happens when you've got a speed step down (e.g a 1Gb interface connected to a 100Mb interface). If your interfaces are configured to auto negotiate their speed and duplex, check them all to ensure that they've negotiated their settings correctly.

Related Topic