I am running my custom service, let's call it "foo.service".
It's a gunicorn python process which logs a few things.
I used to be able to get a real-time log overview using journalctl -u <service> -f
but now the journal seems to be stuck at some past logs. When I use systemctl status <service>
it does show the latest logs though. So my logging does work, but journalctl seems to be stuck and doesn't show any updates.
Example:
journalctl -u foo.service -f
Nov 25 16:19:09 <name> systemd[1]: Started Instance to load up the program and its endpoints.
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,844 [INFO]: Connecting to localhost:9773
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,845 [INFO]: Connecting to localhost:9771
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,846 [INFO]: Connected
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,846 [INFO]: Connected
systemctl status foo.service
Nov 26 11:39:53 <name> systemd[1]: Started Instance to load up the program and its endpoints.
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,458 [INFO]: Connecting to localhost:9773
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,459 [INFO]: Connecting to localhost:9771
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,460 [INFO]: Connected
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,460 [INFO]: Connected
The latter is logging that was just created, so that works like a charm, but from my understanding the journalctl seems to be updated as well. This did work in the past, I am encountering these problems since the day before yesterday.
I tried restarting journalctl but that didn't seem to work.
Thanks in advance.
EDIT:
When I restart my service I notice gunicorn threw an exception when I look at the status of the systemctl service, I cannot scroll up so I have no clue as to what the cause of the error is (systemctl status):
Nov 26 15:08:27 <name> gunicorn[29390]: self.log.info("Shutting down: %s", self.master_name)
Nov 26 15:08:27 <name> gunicorn[29390]: File "/opt/my-program/venv/lib/python3.5/site-packages/gunicorn/glogging.py", line 271, in info
Nov 26 15:08:27 <name> gunicorn[29390]: self.error_log.info(msg, *args, **kwargs)
Nov 26 15:08:27 <name> gunicorn[29390]: Message: 'Shutting down: %s'
Nov 26 15:08:27 <name> gunicorn[29390]: Arguments: ('Master',)
Nov 26 15:08:27 <name> systemd[1]: Started Instance to load up the program and its endpoints.
My foo.service configuration file:
[Unit]
Description=Instance to load up the program and its endpoints
After=network.target
[Service]
User=root
Group=www-data
WorkingDirectory=/opt/my-program/my-program-thing
Environment="PATH=/opt/my-program/venv/bin"
ExecStart=/opt/my-program/venv/bin/gunicorn --workers 1 --threads 12 --bind unix:foo.sock -m 007 app:app --bind 0.0.0.0:8085 --access-logfile '/var/log/foo.info.log' --error-logfile '/var/log/foo.err.log'
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Best Answer
Try adding these lines to your service file under the [Service] tag
StandardOutput=journal StandardError=journal
Then do a systemctl daemon-reload, then do a systemctl restart your_service.service
I edit the answer for others to see it easily. Problem was lack of space on /var filesystem so journalctl was not able to save logs to disk and read afterwards, that is why there was a difference between systemctl status and journalctl -fu .