Ubuntu 16.04 – Fix Truncated or Missing Logs with Systemd

journalctljournaldsystemctlsystemdUbuntu

I am using Ubuntu 16.04 server, have created a basic nodejs script and packaged it in a simple systemd service.

Content of api.service at /lib/systemd/system/api.service:

[Unit]
Description=api
After=network.target

[Service]
WorkingDirectory=/var/node/api
ExecStart=/usr/bin/node index.js
Restart=always
RestartSec=5s
StartLimitInterval=60s
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

When I watch the logs with journalctl -u api.service -n -f, it seems that some of my script output lines are not recorded, in particular when the script produces a lot of log lines in a short amount of times.

For example, if the the nodejs script is just a simple for loop that outputs the first 3000 integers, only about 2500 lines seems to be recorded in the logs.

Content of index.js at /var/node/api/index.js:

for (var i = 1; i <= 3000; i++) {
    console.log(i);
}
console.log("end of script");

When viewing the logs with journalctl -u api.service -n -f, I only get about 2500 lines :

May 04 13:52:15 test systemd[1]: Stopping api...
May 04 13:52:15 test systemd[1]: Stopped api.
May 04 13:52:15 test systemd[1]: Started api.
May 04 13:52:15 test node[20993]: 1
May 04 13:52:15 test node[20993]: 2
May 04 13:52:15 test node[20993]: 3
May 04 13:52:15 test node[20993]: 4
May 04 13:52:15 test node[20993]: 5
May 04 13:52:15 test node[20993]: 6
...
...
...
May 04 13:52:16 test node[20993]: 2497
May 04 13:52:16 test node[20993]: 2498
May 04 13:52:16 test node[20993]: 2499
May 04 13:52:16 test node[20993]: 2500
May 04 13:52:16 test node[20993]: 2501

It is also worth noting that the "end of script" is not output either, even if I delay it with a setTimeout.

Any idea what is going wrong here ?

Best Answer

This is happening probably because of rate limiting in journalctl conf. Check RateLimitInterval and RateLimitBurst in /etc/systemd/journald.conf file.