How to analyse vsftpd log files with goaccess

goaccesslog-filesvsftpd

I'm trying to use goaccess log analysis tool, to analyse vsftpd FTP server's logs. I'm aware that being a web server log analyser, goaccess is not the best tool for this. That being said, it's flexible enough with the log format and we're already using it to analyse the web server's log, so I decided to give it a try.

By default vsftpd has a very chatty log:

Mon Mar 23 06:00:00 2020 [pid 11111] CONNECT: Client "1.1.1.1"
Mon Mar 23 06:00:00 2020 [pid 11111] [ftp] OK LOGIN: Client "1.1.1.1", anon password "blablabla"
Mon Mar 23 06:00:00 2020 [pid 11111] [ftp] FAIL DOWNLOAD: Client "1.1.1.1", "/file1", 0.00Kbyte/sec
Mon Mar 23 06:00:00 2020 [pid 11111] [ftp] OK DOWNLOAD: Client "1.1.1.1", "/file2", 17500 bytes, 203.15Kbyte/sec

As modern servers ­—usually— have single a line of log for a particular request, my first step was to change the logging to xferlog format, which gives a single line per transfer.

# /etc/vsftpd.conf
xferlog_enable=YES
xferlog_std_format=YES
xferlog_file=/var/log/vsftpd.log

Now I have log lines like the following.

Sat Mar 28 06:00:00 2020 1 1.1.1.1 17500 /file1 b _ o a anonymous ftp 0 * c
Sat Mar 28 06:00:00 2020 1 1.1.1.1 0 /file2 b _ o a anonymous@host ftp 0 * i

As this is a download server only getting anonymous & passive FTP requests, I can assume the anonymous and anonymous@host are whatever the client provides for anonymous FTP.

How can I analyse this log with goaccess?

Best Answer

As mentioned in the vsftpd documentation, the xferlog format is shared among various server software. proftpd documents have a description of the format.

In a nutshell the fields in each line are called:

current-time transfer-time remote-host file-size filename transfer-type ↩ 
  special-action-flag direction   access-mode username service-name ↩ 
  authentication-method authenticated-user-id completion-status

So your example,

Sat Mar 28 06:00:00 2020 1 1.1.1.1 17500 /file1 b _ o a anonymous ftp 0 * c

corresponds to these fields:

current-time: Sat Mar 28 06:00:00 2020 
transfer-time: 1
remote-host: 1.1.1.1
file-size: 17500
filename: /file1
transfer-type: b (binary)
special-action-flag: _ (no compression)
direction: o (outgoing, this will be always 'o' for a download only server)
access-mode: a (anonymous)
username:  whatever the client provides for anonymous FTP
service-name: ftp
authentication-method: 0 (none)
authenticated-user-id: * (none)
completion-status: c (completed) or i (incomplete)

Now we can try mapping the various log fields to what goaccess accepts, taking a close look at the units, and ignoring anything that is not applicable with the %^ format specifier:

  • The timestamp is problematic, as the time is interleaved between day and year. Fortunately, ignoring the year works ok in goaccess. You may have problems on or around new years. We'll use date-format %a %b %d and time-format %H:%M:%S.
  • transfer-time is logged in seconds, while goaccess only supports micro- or milli-second resolution, so we'll have to ignore it.
  • remote-host is one of the mandatory goaccess options, and is %h
  • file-size maps cleanly to %b in bytes.
  • filename is the raw URL path(%U), as the full http request %r won't be useful
  • username is random junk, but we'll match to %e for completeness
  • completion-status would have been nice to use this as status, but goaccess expects HTTP status codes, so ignored
  • transfer-type, special-action-flag, direction, access-mode, service-name, authentication-method, authenticated-user-id won't be used

The combined config for parsing the xferlog in goaccess will be:

log-format %d %t %^ %^ %h %b %U %^ %^ %^ %^ %e %^ %^ %^ %^
date-format %a %b %d
time-format %H:%M:%S

Save these lines in a file(~/.goaccessrc) and pass it to goaccess with:

goaccess -p ~/.goaccessrc
Related Topic