SNMP – How to Get Per-Process Memory Use from Cisco Switches

cisco-catalystcisco-iossnmp

I'm trying to get per-process memory use from Cisco switches via SNMP.

I've found an article suggesting you can subtract CISCO-PROCESS-MIB.cpmProcExtMemAllocatedRev and CISCO-PROCESS-MIB.cpmProcExtMemFreedRev (Cisco process memory usage) but that doesn't seem to result in sane values.

Often the values are the same (resulting in a zero) and sometimes the freed is greater than the allocated (resulting in a negative number)- though I think this is probably to do with memory being freed between when I pull the allocated results and when I pull the freed results.

The output of show processes memory on a switch shows the same results that I see via SNMP (insane values if the allocated – freed logic is correct), but it also shows a Holding column that looks like it has what I need.

    Switch1#show processes memory
    Processor Pool Total:  175382376 Used:   47922940 Free:  127459436
          I/O Pool Total:   16777216 Used:   13591380 Free:    3185836
    Driver te Pool Total:    4194304 Used:         40 Free:    4194264

     PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
       0   0  118863872   66054580   48365728          0          0 *Init*
       0   0      12476 2411645460      12476          0          0 *Sched*
       0   0 3937788032 4286508576    3540184   15562527    1490354 *Dead*
       0   0          0          0     394476          0          0 *MallocLite*
       1   0     348672     187988     175856          0          0 Chunk Manager
       2   0        232        232       4160          0          0 Load Meter
       3   0         76          0       9236          0          0 hulc_entropy_thr
       4   0          0          0      10080          0          0 Connection Mgr
       5   0       4712       4520      11692          0          0 Check heaps
       6   0   16741228   29720504      38796   14834428   22351307 Pool Manager

I can't find any references to Holding in the CISCO-PROCESS-MIB and I've had no luck searching on the internet.

Does anyone know how to get this Holding field via SNMP?

Best Answer

I've just spent a good half an hour looking at this, and I don't think there is a way to get an accurate number from SNMP.

This is the data available from the 3 tables in cpmProcess (there are many more OIDs, but they don't seem to be populated on the devices i was looking at)

CISCO-PROCESS-MIB::cpmProcessPID.1.5 = Gauge32: 5
CISCO-PROCESS-MIB::cpmProcessName.1.5 = STRING: Pool Manager
CISCO-PROCESS-MIB::cpmProcessuSecs.1.5 = Gauge32: 215 microseconds
CISCO-PROCESS-MIB::cpmProcessTimeCreated.1.5 = Timeticks: (267) 0:00:02.67
CISCO-PROCESS-MIB::cpmProcessAverageUSecs.1.5 = Gauge32: 215 microseconds
CISCO-PROCESS-MIB::cpmProcExtMemAllocated.1.5 = Gauge32: 23223664 bytes
CISCO-PROCESS-MIB::cpmProcExtMemFreed.1.5 = Gauge32: 1784383920 bytes
CISCO-PROCESS-MIB::cpmProcExtInvoked.1.5 = Counter32: 418471
CISCO-PROCESS-MIB::cpmProcExtRuntime.1.5 = Counter32: 90380 microseconds
CISCO-PROCESS-MIB::cpmProcExtUtil5Sec.1.5 = Gauge32: 0
CISCO-PROCESS-MIB::cpmProcExtUtil1Min.1.5 = Gauge32: 0
CISCO-PROCESS-MIB::cpmProcExtUtil5Min.1.5 = Gauge32: 0
CISCO-PROCESS-MIB::cpmProcExtPriority.1.5 = INTEGER: critical(1)
CISCO-PROCESS-MIB::cpmProcExtMemAllocatedRev.1.5 = Gauge32: 23223664 bytes
CISCO-PROCESS-MIB::cpmProcExtMemFreedRev.1.5 = Gauge32: 1784383920 bytes
CISCO-PROCESS-MIB::cpmProcExtInvokedRev.1.5 = Counter32: 418471
CISCO-PROCESS-MIB::cpmProcExtRuntimeRev.1.5 = Counter32: 90380 microseconds
CISCO-PROCESS-MIB::cpmProcExtUtil5SecRev.1.5 = Gauge32: 0 percent
CISCO-PROCESS-MIB::cpmProcExtUtil1MinRev.1.5 = Gauge32: 0 percent
CISCO-PROCESS-MIB::cpmProcExtUtil5MinRev.1.5 = Gauge32: 0 percent
CISCO-PROCESS-MIB::cpmProcExtPriorityRev.1.5 = INTEGER: critical(1)

I wrote a script to snmpget all of the values to prevent timing issues, and sorted them by Allocated - Freed :

(the middle bit removed)

PDU DISPATCHER        3446519088
IP SNMP               1742301596
SNMP ENGINE           792807060
IP Input              635031724
Logger                27247348
TCP Protocols         15053688
Per-Second Jobs       10510488

...

SSM connection manager  0
Crypto IKE Dispatcher  -472
IP Background         -1880
DHCPD Timer           -47424
TTY Background        -103736
LOCAL AAA             -126696
Transport Port Agent  -181900
IP Cache Ager         -1120096
crypto sw pk proc     -2327016
TCP Timer             -606529520
Pool Manager          -1761160256

The processes with negative values don't seem to have much in common, and it's strange that PDU DISPATCHER and IP SNMP would be using the most memory.

I think it might be down to shared memory and the original OIDs pre-dating modern memory management in IOS.

It's possible that the data needed is contained in the OIDs Cisco don't seem to often populate :

1.3.6.1.4.1.9.9.109.1.2.3.1.15 cpmProcessTextSegmentSize
1.3.6.1.4.1.9.9.109.1.2.3.1.16 cpmProcessDataSegmentSize
1.3.6.1.4.1.9.9.109.1.2.3.1.17 cpmProcessStackSize
1.3.6.1.4.1.9.9.109.1.2.3.1.18 cpmProcessDynamicMemorySize

I can't find any other Cisco MIBs that contain better data, so I don't think there is a reliable way to get this information.

Related Solutions

Cisco – how to get scp via snmp to work with cisco routers

I just tried this on my CPE:

[ytti@lintukoto ~]% cat moi2.sh 
#!/bin/sh

snmp="snmpset -v2c -cfoo bu.ip.fi"

$snmp 1.3.6.1.4.1.9.9.96.1.1.1.1.2.9 i 4 \
      1.3.6.1.4.1.9.9.96.1.1.1.1.3.9 i 4 \
      1.3.6.1.4.1.9.9.96.1.1.1.1.4.9 i 1 \
      1.3.6.1.4.1.9.9.96.1.1.1.1.5.9 a 91.198.120.2 \
      1.3.6.1.4.1.9.9.96.1.1.1.1.6.9 s filename \
      1.3.6.1.4.1.9.9.96.1.1.1.1.7.9 s username \
      1.3.6.1.4.1.9.9.96.1.1.1.1.8.9 s password \
      1.3.6.1.4.1.9.9.96.1.1.1.1.14.9 i 4
sleep 10
$snmp 1.3.6.1.4.1.9.9.96.1.1.1.1.14.9 i 6
[ytti@lintukoto ~]%

Which copies running config (4) to network (1), by swapping these you can change the direction (from network to running).

Running above script my home directory will have 'filename' file, which contains my CPE running-config:

[ytti@lintukoto ~]% ls -la filename
ls: cannot access filename: No such file or directory
[2 ytti@lintukoto ~]% ./moi2.sh      
iso.3.6.1.4.1.9.9.96.1.1.1.1.2.9 = INTEGER: 4
iso.3.6.1.4.1.9.9.96.1.1.1.1.3.9 = INTEGER: 4
iso.3.6.1.4.1.9.9.96.1.1.1.1.4.9 = INTEGER: 1
iso.3.6.1.4.1.9.9.96.1.1.1.1.5.9 = IpAddress: 91.198.120.2
iso.3.6.1.4.1.9.9.96.1.1.1.1.6.9 = STRING: "filename"
iso.3.6.1.4.1.9.9.96.1.1.1.1.7.9 = STRING: "username"
iso.3.6.1.4.1.9.9.96.1.1.1.1.8.9 = STRING: "password"
iso.3.6.1.4.1.9.9.96.1.1.1.1.14.9 = INTEGER: 4
iso.3.6.1.4.1.9.9.96.1.1.1.1.14.9 = INTEGER: 6
[ytti@lintukoto ~]% ls -la filename
-rw-r--r-- 1 ytti ytti 16172 Jun 11 00:35 filename
[ytti@lintukoto ~]%

On top of what @daniel mentions also your '14' or 'rowstatus' is wrong, you use 1 'active', while you should use 4 'createAndGo'.

Cisco – Process Memory Usage

I don't have a device with an ipfib process available, but assuming you're running IOS, the principles are the same no matter what the process name is...

Find the index for the process in question:

First, walk cpmProcessName and find the index... for instance, I'll look for "ARP Input"...

(py27_test)[mpenning@tsunami fast]$ snmpbulkwalk -v 2c -c public 10.1.1.1  \
  .1.3.6.1.4.1.9.9.109.1.2.1.1.2 | grep "ARP Input"
CISCO-PROCESS-MIB::cpmProcessName.1.12 = STRING: ARP Input
CISCO-PROCESS-MIB::cpmProcessName.1.239 = STRING: RARP Input
(py27_test)[mpenning@tsunami fast]$

We can see the index for the "ARP Input" process is 1.12...

All memory allocated:

Now let's find all the memory that was ever allocated to this process. We will do an snmpget on cpmProcExtMemAllocatedRev... be sure to append 1.12 to the cpmProcExtMemAllocatedRev OID...

(py27_test)[mpenning@tsunami fast]$ snmpget -v 2c -c public 10.1.1.1 \
  .1.3.6.1.4.1.9.9.109.1.2.3.1.1.1.12
CISCO-PROCESS-MIB::cpmProcExtMemAllocatedRev.1.12 = Gauge32: 1259940452 bytes
(py27_test)[mpenning@tsunami fast]$

cpmProcExtMemAllocatedRev is the sum of all memory ever allocated to the process; that means you'll have to subtract all memory that was ever freed from the process.

All memory freed:

To find all the memory that was ever freed, poll the cpmProcExtMemFreedRev OID, and append the index (1.12)...

(py27_test)[mpenning@tsunami fast]$ snmpget -v 2c -c public 10.1.1.1 \
  .1.3.6.1.4.1.9.9.109.1.2.3.1.2.1.12
CISCO-PROCESS-MIB::cpmProcExtMemFreedRev.1.12 = Gauge32: 1259931892 bytes
(py27_test)[mpenning@tsunami fast]$

Totals / math:

So the total memory used by my "ARP Input" process is:

AllocatedMem - FreedMem   = MemoryUsed

1259940452   - 1259931892 = *8560 Bytes*