Cisco – Input packet drops on port channel

ciscotroubleshooting

I have a certain number of packets dropped on my port channel interface with 3gbps bundled speed, on Cisco 6500 with Sup 720. You can see underneath the number of dropped packet within a minute, after I cleared the counters.

I'm wondering why packets don't enter in the queue at all before being dropped. Please help me resolve the problem with queue drops.

Will adding another interface in to the bundle resolve help reduce the drops?

CORESW01#show interfaces port-channel 50                      
Port-channel50 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is XXXXXXXXXX (bia XXXXXXXX)
  Description: [***Etherchannel Connected to DELL-ENCLOSURE-B1 Ports 17,18,19,20***]
  MTU 1500 bytes, BW 3000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s
  input flow-control is off, output flow-control is off
  Members in this channel: Gi2/33 Gi2/34 Gi2/35 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 17:30:41
  --THIS LINE--> Input queue: 0/2000/73599/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 7422000 bits/sec, 1105 packets/sec
  5 minute output rate 6306000 bits/sec, 1329 packets/sec
     77713779 packets input, 87465246992 bytes, 0 no buffer
     Received 530784 broadcasts (258881 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 5 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     46002614 packets output, 24633769164 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

11-18-2016

CORESW01#show interfaces gigabitEthernet 2/33  
GigabitEthernet2/33 is up, line protocol is up (connected)  
  Hardware is C6k 1000Mb 802.3, address is XX  
  Description: [***Uplink Connected to DELL-ENCL1-B2***]  
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,   
     reliability 255/255, txload 2/255, rxload 1/255  
  Encapsulation ARPA, loopback not set  
  Keepalive set (10 sec)  
  Full-duplex, 1000Mb/s  
  input flow-control is off, output flow-control is off  
  Clock mode is auto  
  ARP type: ARPA, ARP Timeout 04:00:00  
  Last input never, output 00:00:29, output hang never   
  Last clearing of "show interface" counters 04:47:46  
  Input queue: 0/2000/359/0 (size/max/drops/flushes); Total output drops: 0  
  Queueing strategy: fifo  
  Output queue: 0/40 (size/max)  
  5 minute input rate 3744000 bits/sec, 820 packets/sec  
  5 minute output rate 10958000 bits/sec, 1649 packets/sec  
     7916371 packets input, 4017248800 bytes, 0 no buffer  
     Received 32074 broadcasts (24595 multicasts)  
     0 runts, 0 giants, 0 throttles  
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored  
     0 watchdog, 0 multicast, 0 pause input  
     0 input packets with dribble condition detected  
     13081762 packets output, 4307087164 bytes, 0 underruns  
     0 output errors, 0 collisions, 0 interface resets  
     0 babbles, 0 late collision, 0 deferred  
     0 lost carrier, 0 no carrier, 0 PAUSE output  
     0 output buffer failures, 0 output buffers swapped out  

CORESW01#show interfaces gigabitEthernet 2/34  
GigabitEthernet2/34 is up, line protocol is up (connected)  
  Hardware is C6k 1000Mb 802.3, address is XX  
  Description: [***Uplink Connected to DELL-ENCL1-B2***]  
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,   
     reliability 255/255, txload 1/255, rxload 1/255  
  Encapsulation ARPA, loopback not set  
  Keepalive set (10 sec)  
  Full-duplex, 1000Mb/s  
  input flow-control is off, output flow-control is off  
  Clock mode is auto   
  ARP type: ARPA, ARP Timeout 04:00:00  
  Last input never, output 00:00:36, output hang never  
  Last clearing of "show interface" counters 04:47:49  
  Input queue: 0/2000/136/0 (size/max/drops/flushes); Total output drops: 0  
  Queueing strategy: fifo  
  Output queue: 0/40 (size/max)    
  5 minute input rate 6338000 bits/sec, 647 packets/sec  
  5 minute output rate 821000 bits/sec, 298 packets/sec  
     7605420 packets input, 6914668706 bytes, 0 no buffer  
     Received 79343 broadcasts (26075 multicasts)  
     0 runts, 0 giants, 0 throttles  
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored  
     0 watchdog, 0 multicast, 0 pause input  
     0 input packets with dribble condition detected  
     11489585 packets output, 5625019559 bytes, 0 underruns  
     0 output errors, 0 collisions, 0 interface resets  
     0 babbles, 0 late collision, 0 deferred  
     0 lost carrier, 0 no carrier, 0 PAUSE output  
     0 output buffer failures, 0 output buffers swapped out  

CORESW01#show interfaces gigabitEthernet 2/35  
GigabitEthernet2/35 is up, line protocol is up (connected)  
  Hardware is C6k 1000Mb 802.3, address is XX  
  Description: [***Uplink Connected to DELL-ENCL1-B2***]  
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,   
     reliability 255/255, txload 1/255, rxload 1/255  
  Encapsulation ARPA, loopback not set  
  Keepalive set (10 sec)  
  Full-duplex, 1000Mb/s  
  input flow-control is off, output flow-control is off  
  Clock mode is auto  
  ARP type: ARPA, ARP Timeout 04:00:00  
  Last input never, output 00:00:44, output hang never  
  Last clearing of "show interface" counters 04:48:08  
  Input queue: 0/2000/193/0 (size/max/drops/flushes); Total output drops: 0  
  Queueing strategy: fifo 
  Output queue: 0/40 (size/max)  
  5 minute input rate 4163000 bits/sec, 515 packets/sec  
  5 minute output rate 2445000 bits/sec, 373 packets/sec  
     25776481 packets input, 32767081104 bytes, 0 no buffer  
     Received 37253 broadcasts (29829 multicasts) 
     0 runts, 0 giants, 0 throttles  
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored  
     0 watchdog, 0 multicast, 0 pause input  
     0 input packets with dribble condition detected  
     14263612 packets output, 13067998250 bytes, 0 underruns  
     0 output errors, 0 collisions, 0 interface resets  
     0 babbles, 0 late collision, 0 deferred  
     0 lost carrier, 0 no carrier, 0 PAUSE output  
     0 output buffer failures, 0 output buffers swapped out  



CORESW01#show module   
Mod Ports Card Type                              Model              Serial No.  
--- ----- -------------------------------------- ------------------ -----------
  1   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       XX  
  2   48  48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-45AF  Xx  
  5    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       SX  

Mod MAC addresses                       Hw    Fw           Sw           Status  
--- ---------------------------------- ------ ------------ ------------ -------
  1  0016.XXXX.7ad6 to 0016.XXXX.7aed   2.3   12.2(14r)S5  12.2(18)SXF3 Ok  
  2  0016.XXXX.f2e0 to 0016.XXXX.f30f   2.0   8.4(1)       8.5(0.46)RFW Ok  
  5  0014.XXXX.d988 to 0014.XXXX.d98b   4.5   8.1(3)       12.2(18)SXF3 Ok  

Mod  Sub-Module                  Model              Serial       Hw     Status   
---- --------------------------- ------------------ ----------- ------- -------
  1  Centralized Forwarding Card WS-F6700-CFC       XX  2.0    Ok  
  2  IEEE Voice Daughter Card    WS-F6K-48-AF       XX  2.0    Ok  
  5  Policy Feature Card 3       WS-F6K-PFC3B       XX  2.2    Ok  
  5  MSFC3 Daughterboard         WS-SUP720          XX  2.4    Ok  

Mod  Online Diag Status   
---- -------------------
  1  Pass  
  2  Pass  
  5  Pass  
CORE-SW01#


Group: 50  
---------- 
                Port-channels in the group:    
        ----------------------

Port-channel: Po50    (Primary Aggregator)  

------------

Age of the Port-channel   = 389d:22h:28m:24s  
Logical slot/port   = 14/2          Number of ports = 3  
Port state          = Port-channel Ag-Inuse   
Protocol            =   LACP  

Ports in the Port-channel:   

Index   Load   Port     EC state        No of bits 
------+------+------+------------------+----------- 
  0     49     Gi2/33   Active    3  
  1     92     Gi2/34   Active    3  
  2     24     Gi2/35   Active    2  

Time since last port bundled:    389d:22h:27m:19s    Gi2/35

Since I am not seeing the queue count jump, I think this could be because of bursts of traffic.

Best Answer

You get a high number of input drops if your device cannot service the input queue fast enough. There can be many reasons for this. For instance, routing policies or other services which cause process switching will slow your CPU and increase the number of input drops.

Cisco has a document, Troubleshooting Input Queue Drops and Output Queue Drops, which discusses the problem:

Input Queue Drops

When a packet enters the router, the router attempts to forward it at interrupt level. If a match cannot be found in an appropriate cache table, the packet is queued in the input queue of the incoming interface to be processed. Some packets are always processed, but with the appropriate configuration and in stable networks, the rate of processed packets must never congest the input queue. If the input queue is full, the packet is dropped.

Here is a sample output:

router#show interfaces ethernet 0/0 
...
Input queue: 30/75/187/0 (size/max/drops/flushes); Total output drops: 0 
Output queue :0/40 (size/max)...

In this sample output, there is no way to see exactly which packets have been dropped. In order to troubleshoot input queue drops, you must find out which packets fill the input queue. In this example, 30 packets are in the input queue of interface ethernet0/0 when the show interfaces ethernet 0/0 command is issued. The queue depth is 75 packets and there have been 187 drops since the interface counters were last cleared.

The system counts input queue drops if the number of packet buffers allocated to the interface is exhausted or reaches its maximum threshold. You can increase the maximum queue value with the hold-queue command for each interface (the queue length value can be between 0 and 4096. The default value is 75).

Note: Shared-memory routers (1600, 2500, and 4000 series), also use the input queue for fast-switched traffic. If you get input queue drops on those platforms, ensure that all traffic uses the best switching path available (see Performance Tuning Basics). Input queue drops generally occur when a packet is process-switched. A process switch means that the router cannot use a preferable route-cache method, such as fast switching or Cisco Express Forwarding (CEF), to handle the forwarding decision. If input drops are still present, it implies that there is simply too much traffic. Consider a hardware upgrade, or try to decrease the traffic load.

These are the conditions for input queue drop counter. They usually occur when the router receives bursty traffic and cannot handle all packets.

  • The Rx FIFO which is accessible by the interface PHY and interface DMA is full and any new frames that arrive in this condition will be
    dropped (normally called as overflow) and the rx_overflow counter
    (seen through show controller interface-id) will be
    incremented. When rx_overflow counter is incremented by one, it
    indicates that overflow condition has occurred once and is not
    indicative of the number of frames dropped.
  • The Rx ring which is accessible by the interface DMA and interface driver code is full. Any new frame transfers from the DMA cannot
    proceed with this condition, since there are no free entries in Rx
    ring and hence the frames sent are dropped (termed as overrun
    condition). The rx_int_drop counter (seen through show controller interface-id) is also incremented by one. Again, if rx_int_drop is incremented by one it indicates that there is one occurrence of an
    overrun condition, and the number of frames dropped is not known.

The input hold queue size can be increased from the default 75 packets. The hold queue stores packets received from the network that wait to be sent to the client. Cisco recommends that the queue size not exceed ten packets on asynchronous interfaces. For most other interfaces, queue length must not exceed 100. The input hold queue prevents a single interface from flooding the network server with too many input packets. Further input packets are discarded if the interface has too many input packets outstanding in the system.

Router(conf-if)# hold-queue length in

For Catalyst Switches, Cisco recommends to make this adjustment on all L3 interfaces on the device, both physical interfaces and VLAN interfaces. L2 ports configured with the switchport command can be left at the default value.

Note: After you apply this command, you need to clear the interface counters and then monitor the network.

Caution: An increase in the hold queue can have detrimental effects on network routing and response times. For protocols that use SEQ/ACK packets to determine round-trip times, do not increase the output queue. Dropping packets instead informs hosts to slow down transmissions to match available bandwidth. This is generally better than duplicate copies of the same packet within the network, which can happen with large hold queues.

Related Topic