Cisco TCP Syslog – Problems with TCP-Syslog on Cisco Devices

You're right, you wouldn't really see the burstiness easily on SNMP. 1GE can send 1.48Mpps, so it takes very very little time to congest the the 45Mbps, which can handle less than 75kpps.

If your ingress is 1GE and egress is 45Mbps, then obviously the congestion point of 45Mbps will need to drop packets. This is normal and expected. If you increase buffers you'll introduce more delay.
1GE takes 0.45ms to send 40 1500B IP frames, which is right now the amount of burst you can handle. However dequeueing them on the 45Mbps already takes 10ms.

If you don't have any acute problem, I would probably not do anything about it. But if some traffic is more eligible for dropping than other, then you should replace FIFO with class-based queueing. Say maybe you want to prioritize so that more ftp is dropped and less voip.
Then it'll also make more sense to add more buffering on the ftp traffic, as it's not really sensitive to delay.

If you want to try your luck with deeper buffers, something like this should suffice:

policy-map WAN-OUT
 class class-default
    fair-queue
    queue-limit 200 packets
!
interface Serial1/0
  service-policy output WAN-OUT

This would cause 50ms buffers on the Serial1 and would allow you to handle up-to 2.25ms burst from single Gige interface.

Summary

You should use Cisco's Embedded Syslog Manager. ESM can dynamically modify or throttle syslog messages when they are generated on the router.

ESM Demo

I built an example (see bottom of answer) of how to rate-limit configuration messages within a test time window; for the purposes of this demo, I substituted [regexp {CONFIG} $::orig_msg] instead of [regexp {FAN_LOW_RPM} $::orig_msg] so I could illustrate rate-limiting messages like %SYS-5-CONFIG_I: Configured from console by vty0.

I edited the tcl script at the bottom of the answer with [regexp {CONFIG} $::orig_msg], and tftp'd the script into flash...

DEN-EDGE-02#copy tftp://172.16.1.5/filterSyslog.tcl flash:
Destination filename [filterSyslog.tcl]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]
Accessing tftp://172.16.1.5/filterSyslog.tcl...
Loading filterSyslog.tcl from 172.16.1.5: !
[OK - 684 bytes]

684 bytes copied in 0.104 secs (6577 bytes/sec)
DEN-EDGE-02#

Then I configured my router with the name of the script, and the syslog server's address (172.16.1.5).

logging filter flash:filterSyslog.tcl
logging trap debugging
logging host 172.16.1.5 vrf mgmtVrf filtered

Now when you go into configuration mode on the router, the syslog messages are rate-limited.

[mpenning@tsunami tftpboot]$ sudo tshark -ni eth0 udp and host 172.16.1.204
Running as user "root" and group "root". This could be dangerous.
Capturing on eth0
  3.472614 172.16.1.204 -> 172.16.1.5   Syslog 177 LOCAL7.NOTICE: 278: Apr 21 05:37:58.189
   CDT: %SYS-5-CONFIG_I: Configured from console by vty0 (172.16.1.5) - This message was
   rate-limited by ESM

How it works

The ESM script at the bottom of the answer rate-limits FAN_LOW_RPM messages. The example leverages the fact that NVMON-4-FAN_LOW_RPM messages are sent every 30 seconds. For simplicity, I use an absolute 30-second window between 23:59:30 and 23:59:59 to rate-limit the messages. This script assumes the syslogs are sent at a constant rate, and are not intermittent. In the attached script, I use timestamps in HHMMSS (24-hour) format so they would map easily to integers.

When a syslog message is ready to be sent, IOS stores it in $::orig_msg. I just built a quick series of if .. else clauses to detect whether the syslog message:

• Matches the regular expression (in this case, FAN_LOW_RPM)
• Occurs in the 30-second window between 23:59:30 and 23:59:59 (inclusive)

If the message contains FAN_LOW_RPM and is within the time window, the script sends the message. Other messages containing FAN_LOW_RPM messages are not sent. All other syslogs are sent (because we only want to silence the noisy messages).

FYI, for simplicity I intentionally avoided persisting timestamp values between the last NVMON-4-FAN_LOW_RPM syslog message seen, although someone could do that too.

ESM syslog rate-limit script:

Save this file in flash as flash:filterSyslog.tcl

## Filename: filterSyslog.tcl
proc forceInteger { x } {
    set count [scan $x %d%s n rest]
    if { $count <= 0 || ( $count == 2 && ![string is space $rest] ) } {
        # This is an error
        return "-1"
    }
    return $n
}
set time_start 235900
set time_end 235959
# See http://wiki.tcl.tk/498 for information about TCL's strange number-handling
set timestamp [forceInteger [clock format [clock seconds] -format %H%M%S]]

### Modify the regexp below and use any message you want to rate-limit...
if { [regexp {FAN_LOW_RPM} $::orig_msg] } {
    if {($time_start <= $timestamp) && ($timestamp <= $time_end)} {
        # Send syslog messages inside the $time_start and $time_end
        return "$::orig_msg - This message was rate-limited by ESM"
    } else {
        # Drop syslog messages outside $time_start to $time_end
        return ""
    }
} else {
    # Return all other syslog messages as usual
    return $::orig_msg
}