Log Rotation Causing nxlog Failure

logginglogstashrotation

I'm using nxlog to watch two vCenter log files on a Windows Server 2008 system and I have a very simple config to ship messages to Logstash. Nxlog began failing yesterday after a file was rotated. Here's my config:

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input in1>
    Module      im_file
    File        "C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd-[0-5][0-9].log"
    SavePos     TRUE
</Input>

<Input in2>
    Module      im_file
    File        "C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd-alert-[0-5][0-9].log"
    SavePos     TRUE
</Input>

<Processor buffer1>
    Module  pm_buffer
    MaxSize 1024
    Type    Mem
    WarnLimit   512
</Processor>

<Processor buffer2>
    Module  pm_buffer
    MaxSize 1024
    Type    Mem
    WarnLimit   512
</Processor>

<Output out1>
    Module      om_udp
    Host        <ip>
    Port        514
</Output>

<Output out2>
    Module      om_udp
    Host        <ip>
    Port        514
</Output>

<Route 1>
    Path        in1 => buffer1 => out1
</Route>

<Route 2>
    Path        in2 => buffer2 => out2
</Route>

vCenter log rotation is a bit odd so I'm thinking that it's at least partially causing this problem. You can see from above the filename I'm watching. This file is created as vpxd-01.log and increments up from there. The files are rotated when they hit 50MB and old logs get compressed after two new files are created, e.g. vpxd-03 is created and vpxd-01 gets compressed.

But yesterday the first file that nxlog had started watching (since being installed last week) was rotated, causing this error:

2013-12-28 19:41:08 WARNING input file does not exist: C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd-47.log

It repeats 12 times across the next 24 hours and then nxlog failed entirely until I restarted the service yesterday morning.

Hopefully my explanation makes sense. I ended up losing about six hours of logs so I want to make sure this doesn't happen again. Does anybody have any solution or suggestions?

Best Answer

Try using this input instead instead of hard defining [0-5][0-9]:

<Input in2>
    Module      im_file
    File        "C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd-*.log"
    SavePos     TRUE
</Input>