Rsyslog Creating Multiple Log Files from Remote Cisco Switch – Issue Resolution

centosciscolinuxrsyslogswitch

So here is our setup

Server: rsyslog server – CentOS 7

Client: Cisco Catalyst C6880-X-LE

/etc/rsyslog.conf from the CentOS 7 server:

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal

$template TmplAuth, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"
authpriv.* ?TmplAuth
*.info;mail.none;authpriv.none;cron.none ?TmplMsg
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log

For some reason in the /var/log/client_logs folder, the cisco logs are creating a new .log file every single time a new log message is generated on the switch. This is obviously not ideal and I want rsyslog to put all of the logs in a single file, and I plan on letting logrotate deal with creating a new log file every day.

Here is an example of what I am seeing in the /var/log/client_logs directory

-rw------- 1 root root 184 Oct 13 14:30 156598.log
-rw------- 1 root root 164 Oct 13 14:30 156599.log
-rw------- 1 root root 186 Oct 13 14:30 156600.log
-rw------- 1 root root 162 Oct 13 14:30 156601.log
-rw------- 1 root root 184 Oct 13 14:30 156602.log
-rw------- 1 root root 164 Oct 13 14:35 156603.log
-rw------- 1 root root 186 Oct 13 14:35 156604.log
-rw------- 1 root root 162 Oct 13 14:35 156605.log
-rw------- 1 root root 184 Oct 13 14:35 156606.log
-rw------- 1 root root 164 Oct 13 14:35 156607.log
-rw------- 1 root root 186 Oct 13 14:35 156608.log
-rw------- 1 root root 162 Oct 13 14:35 156609.log
-rw------- 1 root root 184 Oct 13 14:35 156610.log
-rw------- 1 root root 162 Oct 13 14:39 156611.log
-rw------- 1 root root 164 Oct 13 14:41 156612.log
-rw------- 1 root root 186 Oct 13 14:41 156613.log
-rw------- 1 root root 162 Oct 13 14:41 156614.log
-rw------- 1 root root 184 Oct 13 14:41 156615.log

And this keeps going forever considering it is creating a .log for each new message sent from the Cisco switch. Here is an example of the contents of one of these log files

2021-10-13T14:41:10.866435-07:00 X 156613: X-Switch: .Oct 13 13:40:44 PST: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface GigabitEthernet195/1/0/11, changed state to down

I am not sure if the issue is with the config on the Cisco switch or my rsyslog config, but I have done this before and never had any issues with new .log files being created for every single message sent to the server.

Here is the config on the Cisco switch side

Switch1#show run | include logging
logging userinfo
logging reload debugging
logging event link-status default
logging origin-id hostname
logging host 10.1.1.1
 logging synchronous
 logging synchronous

FYI the logging host statement is the IP address of the CentOS 7 rsyslog server. All of the other logs being captures from other hosts are fine and are not creating new .log files per every message received, but the other systems sending logs are not Cisco switches, they are all various flavors of Linux (primarily CentOS and RHEL).

Any idea why rsyslog is creating new .log files for every message it receives from this Cisco Catalyst switch?

Best Answer

Example of message is follow:

2021-10-13T14:41:10.866435-07:00 X 156613: X-Switch: .Oct 13 13:40:44 PST: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface GigabitEthernet195/1/0/11, changed state to down

Seems that PROGRAMNAME points to the number which is incremented by one with every message, in this specific case 156613:

$template TmplAuth, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"

So config above looks correct. Root cause is probably PROGRAMNAME which is incremented in the message.