Linux journalctl not synced with systemctl status / Journalctl not updating

gunicornjournalctljournaldlinux

I am running my custom service, let's call it "foo.service".
It's a gunicorn python process which logs a few things.

I used to be able to get a real-time log overview using journalctl -u <service> -f but now the journal seems to be stuck at some past logs. When I use systemctl status <service> it does show the latest logs though. So my logging does work, but journalctl seems to be stuck and doesn't show any updates.

Example:

journalctl -u foo.service -f

Nov 25 16:19:09 <name> systemd[1]: Started Instance to load up the program and its endpoints.
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,844 [INFO]: Connecting to localhost:9773
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,845 [INFO]: Connecting to localhost:9771
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,846 [INFO]: Connected
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,846 [INFO]: Connected

systemctl status foo.service

Nov 26 11:39:53 <name> systemd[1]: Started Instance to load up the program and its endpoints.
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,458 [INFO]: Connecting to localhost:9773
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,459 [INFO]: Connecting to localhost:9771
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,460 [INFO]: Connected
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,460 [INFO]: Connected

The latter is logging that was just created, so that works like a charm, but from my understanding the journalctl seems to be updated as well. This did work in the past, I am encountering these problems since the day before yesterday.

I tried restarting journalctl but that didn't seem to work.

Thanks in advance.

EDIT:
When I restart my service I notice gunicorn threw an exception when I look at the status of the systemctl service, I cannot scroll up so I have no clue as to what the cause of the error is (systemctl status):

Nov 26 15:08:27 <name> gunicorn[29390]:     self.log.info("Shutting down: %s", self.master_name)
Nov 26 15:08:27 <name> gunicorn[29390]:   File "/opt/my-program/venv/lib/python3.5/site-packages/gunicorn/glogging.py", line 271, in info
Nov 26 15:08:27 <name> gunicorn[29390]:     self.error_log.info(msg, *args, **kwargs)
Nov 26 15:08:27 <name> gunicorn[29390]: Message: 'Shutting down: %s'
Nov 26 15:08:27 <name> gunicorn[29390]: Arguments: ('Master',)
Nov 26 15:08:27 <name> systemd[1]: Started Instance to load up the program and its endpoints.

My foo.service configuration file:

[Unit]
Description=Instance to load up the program and its endpoints
After=network.target

[Service]
User=root
Group=www-data
WorkingDirectory=/opt/my-program/my-program-thing
Environment="PATH=/opt/my-program/venv/bin"
ExecStart=/opt/my-program/venv/bin/gunicorn --workers 1 --threads 12 --bind unix:foo.sock -m 007 app:app --bind 0.0.0.0:8085 --access-logfile '/var/log/foo.info.log' --error-logfile '/var/log/foo.err.log'
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Best Answer

Try adding these lines to your service file under the [Service] tag

StandardOutput=journal StandardError=journal

Then do a systemctl daemon-reload, then do a systemctl restart your_service.service

I edit the answer for others to see it easily. Problem was lack of space on /var filesystem so journalctl was not able to save logs to disk and read afterwards, that is why there was a difference between systemctl status and journalctl -fu .

Related Topic